Copyright 2011, 2012, Oracle and/or its affiliates. All rights reserved.,.,,,,,,,,,,,,.,...,. U.S. GOVERNMENT END USERS. Oracle programs, including an

Transcription

1 SPARC SuperCluster T4-4 : E

2 Copyright 2011, 2012, Oracle and/or its affiliates. All rights reserved.,.,,,,,,,,,,,,.,...,. U.S. GOVERNMENT END USERS. Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government..., -,,. Oracle Corporation. Oracle Java Oracle Corporation /.. Intel Intel Xeon Intel Corporation. SPARC SPARC International, Inc.. AMD, Opteron, AMD AMD Opteron Advanced Micro Devices. UNIX The Open Group. ( ) 3,. Oracle Corporation 3,. Oracle Corporation 3,, @25097

3 SPARC SuperCluster T SPARC SuperCluster T SPARC SuperCluster T Oracle Fiber Channel

4 SPARC SuperCluster T Oracle Solaris Dynamic Intimate Shared Memory Exadata Storage Server SPARC SuperCluster CPU SPARC SuperCluster T SPARC SuperCluster T OCM SPARC SuperCluster T EM Exadata SPARC SuperCluster Exalogic Exalogic Exalogic Dept1_Cluster1 GridLink Dept1_Cluster1 SDP JDBC Exalogic SDP InfiniBand InfiniBand Sun ZFS Storage PDU PDU SPARC SuperCluster T SPARC SuperClusterT

5 Oracle Exadata Storage Oracle Exadata Storage IP ( ) SPARC SuperCluster T4-4 Oracle Exadata Storage

6 6

7 ,, Oracle SPARC SuperCluster T4-4.,. Oracle. Oracle. SPARC SuperCluster T SPARC SuperCluster T4-4 SPARC T4-4. /opt/oracle/node/doc/e21659_01/index.html Oracle 7

8 . My Oracle Support : Oracle 8 SPARC SuperClusterT

9 SPARC SuperCluster T SPARC SuperCluster T SPARC SuperCluster T SPARC SuperCluster T SPARC SuperClusterT4-4 SPARC SuperCluster T4-4. SPARC SuperCluster T4-4, SPARC SuperCluster T4-4 Oracle Database 11g Real Application Clusters(Oracle RAC) Oracle Solaris Cluster.,. SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 (Online Transaction Processing, OLTP). Oracle Exadata Storage Server, Oracle SPARC SuperCluster T4-4. Oracle SPARC SuperCluster T4-4., SPARC SuperCluster T4-4. 9

10 SPARC SuperClusterT4-4 SPARC SuperCluster T4-4 Oracle. SPARC SuperCluster T4-4. Oracle Oracle Exadata Storage Server Oracle. Oracle Real Application Clusters Oracle Partitioning SPARC SuperCluster T4-4 SPARC SuperCluster T4-4,, InfiniBand. SPARC SuperCluster T4-4 10GbE( ) 1GbE( ). InfiniBand Exadata Exalogic SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 InfiniBand,,. Oracle SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 Oracle Exadata. SPARC T4-4 10GbE SPARC SuperCluster T4-4. SPARC SuperCluster T GB 15K RPM SAS 1 3TB 7.2K RPM SAS 1 96GB Exadata 1 SPARC SuperClusterT4-4 SPARC SuperCluster T4-4.. SPARC SuperCluster T4-4.. SPARC SuperCluster T4-4 SPARC SuperCluster T Cisco SPARC SuperClusterT

11 SPARC SuperClusterT4-4 1U 48. SPARC SuperCluster T4-4. Oracle,.,. Cisco 4948 CAT5.. Cisco Oracle Exadata Storage SPARC SuperCluster T4-4 Oracle Exadata, SPARC SuperCluster T4-4 Oracle Database(DB) Oracle Exadata. Exadata Storage Server SPARC SuperCluster T4-4 Oracle Exadata, SPARC SuperCluster T4-4 Oracle Exadata. Exadata Storage Server. Oracle Database Oracle Solaris 10. Oracle Oracle Solaris Oracle Solaris 10 Oracle Solaris 11. Oracle Exadata Storage Server, Exadata Storage Server. Exadata Storage Server. Exadata Storage Server. SPARC T4-4. Oracle.. Oracle.,.. InfiniBand. 11

12 SPARC SuperCluster T4-4 InfiniBand. Oracle Database,.,. Oracle Exadata Storage Server USB Exadata Storage Server. USB 100mA. SPARC T4-4 SCSI NFS Exadata Storage Server. FCoE(Fibre Channel Over Ethernet). SPARC SuperCluster T4-4, Oracle Exadata Oracle Exalogic Elastic Cloud InfiniBand. SPARC SuperCluster T4-4, Oracle Exadata Oracle Exalogic Elastic Cloud. SPARC SuperCluster T4-4 InfiniBand SPARC T4-4, Exadata Storage Server Sun ZFS Storage SPARC SuperClusterT

13 1 SPARC SuperCluster T4-4, 1: Exadata Storage Server (6) 2: SPARC T4-4 (4) 13

14 3: Sun ZFS Storage 7320 (2) 4: Sun Datacenter InfiniBand Switch 36 (2) 5: Sun Disk Shelf 6: Cisco Catalyst : Sun Datacenter InfiniBand Switch 36 Oracle Exadata Storage. 213 Oracle Exadata Storage. SPARC SuperCluster T4-4 8 SPARC SuperCluster T4-4 Oracle Exadata Exalogic InfiniBand. 183 SPARC SuperCluster T SPARC SuperClusterT

15 2 SPARC SuperCluster T4-4, 1: Sun ZFS Storage 7320 (2) 2: Sun Datacenter InfiniBand Switch 36 (2) 3: Sun Disk Shelf 15

16 4: Cisco Catalyst : SPARC T4-4 (2) 6: Exadata Storage Server (3) 7: Sun Datacenter InfiniBand Switch 36 Oracle Exadata Storage. 213 Oracle Exadata Storage. SPARC SuperCluster T4-4 8 SPARC SuperCluster T4-4 Oracle Exadata Exalogic InfiniBand. 183 SPARC SuperCluster T4-4. SPARC SuperCluster T SPARC SuperCluster T SPARC T Exadata Storage Server 19 Sun ZFS Storage Sun Datacenter InfiniBand Switch Cisco Catalyst SPARCT4-4 SPARC SuperCluster T4-4 SPARC T4-4. SPARC SuperCluster T4-4 SPARC T4-4. SPARC T4-4. ( 20 SPARC T4-4 ) SPARC T SPARC SuperClusterT

17 SPARCT4-4 SPARC T4-4., SPARC T4-4. SPARC T4-4. SPARC T4-4. AC SPARC T4-4 AC.. SPARC T4-4 6 SSD. SPARC SuperCluster T SPARC T

18 Exadata Storage Server Exadata Storage Server SPARC SuperCluster T4-4 6 Exadata Storage Server. SPARC SuperCluster T4-4 3 Exadata Storage Server. Exadata Storage Server. Exadata Storage Server. 23 Exadata Storage Server. Exadata Storage Server Exadata Storage Server. Exadata Storage Server. Exadata Storage Server 12 SPARC SuperCluster T4-4. SPARC SuperCluster 18 SPARC SuperClusterT

19 Sun Datacenter InfiniBand Switch 36 T4-4 Exadata Storage Server Sun ZFS Storage 7320 SPARC SuperCluster T4-4 Sun ZFS Storage Sun ZFS Storage Sun ZFS Storage Sun Disk Shelf 1 Sun ZFS Storage Sun ZFS Storage 7320 Sun Disk Shelf. 25 Sun ZFS Storage Sun ZFS Storage 7320 Sun ZFS Storage 7320 Sun Disk Shelf. Sun ZFS Storage 7320 Sun Disk Shelf. Sun ZFS Storage Sun Disk Shelf SPARC SuperCluster T (logzillas ) SSD. SPARC SuperCluster T SPARC SuperCluster T4-4. Sun Datacenter InfiniBand Switch 36 SPARC SuperCluster T4-4 Sun Datacenter InfiniBand Switch InfiniBand SPARC T4-4, Exadata Storage Server Sun ZFS Storage

20 Cisco Catalyst 4948 Cisco Catalyst 4948 Cisco Catalyst Cisco Catalyst SPARC SuperCluster T4-4. SPARC SuperCluster T SPARC SuperCluster T SPARC T Exadata Storage Server 25 Sun ZFS Storage SPARCT4-4 4 SPARC T (SPARC T4-4 ) 21 InfiniBand (SPARC T4-4 ) 22 Oracle ILOM (SPARC T4-4 ) 23 1GbE (SPARC T4-4 ) 23 10GbE (SPARC T4-4 ) (SPARCT4-4 ) SPARC T SPARC SuperClusterT

21 InfiniBand 개인 네트워크 물리적 연결(SPARC T4-4 서버) 그림 3 SPARC T4-4 서버의 카드 및 포트 위치 그림 범례 1: NET MGT 포트 - Oracle ILOM 관리 네트워크 연결용( 22 페이지 Oracle ILOM 관리 네트워크 물리적 연결(SPARC T4-4 서버) 참조) 2: NET0-NET3 포트 - 1GbE 호스트 관리 네트워크 연결용( 23 페이지 1GbE 호스트 관리 네트워크 물리적 연결(SPARC T4-4 서버) 참조) 3: 이중 포트 InfiniBand 호스트 채널 어댑터 - InfiniBand 네트워크 연결용( 21 페이지 InfiniBand 개인 네트워크 물리적 연결(SPARC T4-4 서버) 참조) 4: 이중 포트 10GbE 네트워크 인터페이스 카드 - 10GbE 클라이언트 액세스 네트워크 연결용( 23 페이지 10GbE 클라이언트 액세스 네트워크 물리적 연결(SPARC T4-4 서버) 참조) InfiniBand 개인 네트워크 물리적 연결(SPARC T4-4 서버) 각 SPARC T4-4 서버에는 이중 포트 Sun QDR InfiniBand PCIe 편평 호스트 채널 어댑터(HCA)가 네 개 포함되어 있습니다. InfiniBand HCA는 모든 SPARC T4-4 서버에서 다음 익스프레스 모듈 포트에 설치됩니다. 익스프레스 모듈 슬롯 1 익스프레스 모듈 슬롯 7 익스프레스 모듈 슬롯 9 익스프레스 모듈 슬롯 12 InfiniBand HCA 위치에 대한 자세한 내용은 20 페이지 카드 및 포트 위치(SPARC T4-4 서버) 를 참조하십시오. 21

22 Oracle ILOM (SPARCT4-4 ) InfiniBand HCA ( 1 2) SPARC T4-4. SPARC T4-4 InfiniBand. InfiniBand. SPARC T4-4 SPARC T4-4 InfiniBand. SPARC T4-4 InfiniBand IP. 29 SPARC SuperCluster T4-4. Oracle ILOM (SPARCT4-4 ) SPARC T4-4 SPARC T4-4 Oracle ILOM (NET MGT ) Oracle ILOM(Integrated Lights Out Manager). SPARC T4-4 Oracle ILOM IP. 22 SPARC SuperClusterT

23 InfiniBand (Exadata Storage Server) NET MGT 20 (SPARC T4-4 ). 1GbE (SPARCT4-4 ) SPARC T4-4 SPARC T4-4 1GbE (NET0 - NET3 ) 1GbE. 1GbE. 29 SPARC SuperCluster T4-4. 1GbE 20 (SPARC T4-4 ). 10GbE (SPARCT4-4 ) SPARC T4-4 Sun Dual 10GbE SFP+ PCIe 2.0 (NIC). 10GbE NIC SPARC T GbE NIC 20 (SPARC T4-4 ). 10GbE NIC ( 0 1). 10GbE NIC IPMP. 10GbE NIC IPMP. SPARC T4-4 10GbE. 29 SPARC SuperCluster T4-4. Exadata Storage Server Exadata Storage Server InfiniBand (Exadata Storage Server) 24 Oracle ILOM (Exadata Storage Server) 24 1GbE (Exadata Storage Server) InfiniBand (Exadata Storage Server) Exadata Storage Server Sun QDR InfiniBand PCIe (HCA). InfiniBand 23

24 Oracle ILOM (Exadata Storage Server) HCA. IP InfiniBand IP Exadata Storage Server. InfiniBand HCA Exadata Storage Server. Exadata Storage Server InfiniBand. Oracle ILOM (Exadata Storage Server) Exadata Storage Server Exadata Storage Server Oracle ILOM (NET MGT ) Oracle ILOM. Exadata Storage Server Oracle ILOM IP. 1GbE (Exadata Storage Server) Exadata Storage Server Exadata Storage Server 1GbE (NET 0 ) 1GbE. Exadata Storage Server 1GbE IP. 24 SPARC SuperClusterT

25 InfiniBand (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 Sun ZFS Storage InfiniBand (Sun ZFS Storage 7320 ) 26 Oracle ILOM (Sun ZFS Storage 7320 ) 26 1GbE (Sun ZFS Storage 7320 ) 27 SAS (Sun ZFS Storage 7320 ) 28 (Sun ZFS Storage 7320 ) InfiniBand (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 Sun ZFS Storage 7320 InfiniBand. Sun ZFS Storage 7320 Sun Dual Port 40GB InfiniBand QDR HCA. InfiniBand HCA. IP InfiniBand IP Sun ZFS Storage InfiniBand HCA Sun ZFS Storage Sun ZFS Storage 7320 InfiniBand. 25

26 Oracle ILOM (Sun ZFS Storage 7320 ) Oracle ILOM (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 Sun ZFS Storage 7320 Oracle ILOM. NET0 Oracle ILOM. Oracle ILOM IP. 1GbE (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 Sun ZFS Storage GbE. 1GbE. NET0( 25 ) NET1( 26 ) 1GbE IP. 26 SPARC SuperClusterT

27 SAS (Sun ZFS Storage 7320 ) SAS (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 SAS-2 HBA. Sun Disk Shelf SIM Link In SIM Link Out. Sun Disk Shelf. 1 SAS-2 HBA Sun Disk Shelf SIM Link Out. 2 SAS-2 HBA Sun Disk Shelf SIM Link In. Sun Disk Shelf SAS. 4 Sun ZFS Storage 7320 SAS 1: 1 2: 2 3: Sun Disk Shelf 27

28 (Sun ZFS Storage 7320 ) (Sun ZFS Storage 7320 ) Sun ZFS Storage SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. SPARC T4-4 SPARC T4-4 AC. AC, AC. Exadata Storage Server Exadata Storage Server AC. AC AC. Sun ZFS Storage 7320 Sun ZFS Storage 7320 AC. AC AC. Sun Disk Shelf Sun Disk Shelf AC. AC AC. Sun Datacenter InfiniBand Switch 36 Sun Datacenter InfiniBand Switch 36 AC. AC AC. 28 SPARC SuperClusterT

29 SPARC SuperClusterT4-4 Cisco Catalyst 4948 Cisco Catalyst 4948 AC. AC AC. SPARC SuperClusterT4-4 SPARC SuperCluster T4-4 (LDom),. SPARC T4-4. SPARC SuperCluster T4-4. CPU - 25% - 100%. CPU. CPU SPARC T % - 100%.. SPARC T4-4. PCI - PCI 1-4. PCI. PCI 16 SPARC T4-4. SPARC SuperCluster T4-4 LDom. SPARC SuperCluster T4-4 SPARC T LDom. LDom. Oracle Solaris 10 Oracle Solaris 11 SPARC SuperCluster T GbE 29

30 31 InfiniBand SPARC SuperCluster T4-4 SPARC T GbE (NIC) - 10GbE. InfiniBand (HCA) - InfiniBand... SPARC T4-4. Oracle Integrated Lights Out Manager(ILOM) SPARC T4-4 Oracle ILOM. SPARC T GbE SPARC T4-4 1GbE (NET0 - NET3).. SPARC T4-4 1GbE (NET0 - NET3) IP (IPMP). IPMP SPARC T GbE 10GbE SPARC T GbE NIC. 10GbE NIC SPARC T SPARC SuperClusterT

31 InfiniBand InfiniBand InfiniBand InfiniBand SPARC T4-4, Sun ZFS Storage 7320 Exadata Storage Server. SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 SPARC T4-4 SPARC T4-4 InfiniBand. SPARC T4-4 InfiniBand SPARC T InfiniBand 32 InfiniBand InfiniBand SPARC T4-4 InfiniBand. SPARC T4-4 Sun Datacenter InfiniBand Switch 36 SPARC T4-4 Exadata Storage Server, Sun Datacenter InfiniBand Switch 36 SPARC T4-4 Sun ZFS Storage 7320, Sun Datacenter InfiniBand Switch 36 SPARC T4-4 InfiniBand HCA. InfiniBand HCA SPARC T4-4. InfiniBand HCA InfiniBand. Oracle Solaris 10 Sun ZFS Storage 7320 InfiniBand 1 Oracle Database 11g Real Application Clusters(Oracle RAC) Exadata Storage Server InfiniBand 1 InfiniBand HCA SPARC T4-4 Sun Datacenter InfiniBand Switch 36. SPARC T InfiniBand (SPARC T4-4 ). 31

32 InfiniBand SPARC T4-4 (Oracle Solaris 10 Oracle Solaris 11 ) InfiniBand. SPARC T4-4 Sun Datacenter InfiniBand Switch 36 SPARC T4-4 Sun ZFS Storage 7320, Sun Datacenter InfiniBand Switch 36 Exadata Storage Server. SPARC T4-4 InfiniBand HCA. InfiniBand HCA SPARC T4-4. InfiniBand HCA InfiniBand. Sun ZFS Storage 7320 InfiniBand 1 Oracle Solaris Cluster InfiniBand 2 InfiniBand HCA SPARC T4-4 Sun Datacenter InfiniBand Switch 36. SPARC T InfiniBand (SPARC T4-4 ).. SPARC SuperCluster T4-4 SPARC T4-4 SPARC T SPARC SuperClusterT

33 Oracle Solaris SPARC SuperCluster T Oracle Solaris Oracle Database 11g Real Application Clusters(Oracle RAC) Oracle. Oracle RAC SPARC T4-4 Oracle Clusterware. Oracle Clusterware Oracle. Oracle Clusterware Oracle RAC. Oracle Clusterware Oracle RAC. Oracle Oracle. Oracle RAC. Oracle RAC.. Oracle RAC Oracle Oracle. Oracle RAC. Oracle RAC Oracle.. Oracle RAC. Oracle Solaris Oracle Solaris Cluster Oracle Solaris. SPARC SuperCluster T4-4 Oracle Solaris Cluster SPARC T4-4 Oracle Solaris. 33

34 SPARC SuperClusterT4-4 SPARC SuperClusterT4-4 SPARC SuperCluster T4-4.. SPARC SuperCluster T4-4 SPARC T4-4 2 Exadata Storage Server 3 Sun ZFS Storage Sun Disk Shelf 1 Sun ZFS Storage Sun Datacenter InfiniBand Switch Cisco Catalyst SPARC T4-4 1 Oracle Solaris 10 1 SPARC T4-4 50% CPU SPARC T4-4 50% SPARC T4-4 PCI Oracle Solaris 10 SPARC T CPU, PCI 35 InfiniBand 36 10GbE CPU, PCI SPARC T4-4. CPU lgroup 34 SPARC SuperClusterT

35 InfiniBand PCI - 0 PCI pci@400 pci@500 InfiniBand InfiniBand SPARC T4-4. SPARCT4-4 SPARC T4-4 4 Sun QDR InfiniBand PCIe HCA 2 SPARC T InfiniBand HCA IPoIB (InfiniBand HCA IPoIB ). InfiniBand 35

36 10GbE HCA SPARC T InfiniBand (Exadata Storage Server). SPARC SuperClusterT4-4 SPARC T4-4 InfiniBand, Exadata Storage Server Sun ZFS Storage 7320 InfiniBand. 10GbE SPARC T4-4 4 Sun Dual 10GbE SFP+ PCIe 2.0 NIC 2 SPARC T SPARC SuperClusterT

37 CPU, PCI 10GbE NIC IPMP. IPMP SPARC SuperCluster T IPMP 10GbE. 2 10GbE NIC GbE NIC - 1 SPARC SuperCluster T4-4 SPARC T4-4 10GbE. Oracle Solaris 10 SPARC T4-4 Oracle Solaris CPU, PCI 38 InfiniBand 40 10GbE CPU, PCI SPARC T4-4 Oracle Solaris 10. CPU lgroup PCI - 1 PCI pci@600 pci@700 CPU, PCI Oracle Solaris

38 InfiniBand InfiniBand InfiniBand SPARC T4-4 Oracle Solaris 10. SPARCT4-4 SPARC T4-4 4 Sun QDR InfiniBand PCIe HCA 2 Oracle Solaris 10 SPARC T Oracle Solaris 10 InfiniBand. 38 SPARC SuperClusterT

39 InfiniBand InfiniBand HCA Oracle Solaris 10 IPoIB. InfiniBand HCA Oracle Solaris Cluster. Oracle Solaris 10 InfiniBand. 7 InfiniBand HCA InfiniBand HCA - 2 SPARC SuperClusterT4-4 SPARC T4-4 Oracle Solaris 10 InfiniBand Sun ZFS Storage 7320 InfiniBand. Oracle Solaris 10, Exadata Storage Server. 39

40 10GbE 10GbE SPARC T4-4 Sun Dual 10GbE SFP+ PCIe 2.0 NIC Oracle Solaris 10 SPARC T GbE NIC IPMP. IPMP SPARC SuperCluster T IPMP Oracle Solaris 10 10GbE. 6 10GbE NIC GbE NIC SPARC SuperClusterT

41 SPARC SuperCluster T SPARC SuperCluster T IP SPARC SuperCluster T4-4 SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320 SPARC T4-4. SPARC T4-4. SPARC T (NET0, NET1, NET2 NET3) Oracle ILOM(Oracle Integrated Lights Out Manager) (NET MGT) 1 InfiniBand Sun QDR InfiniBand PCIe (Host Channel Adapter, HCA) 4 10GbE Sun Dual 10GbE SFP+ PCIe 2.0 (NIC) 4 10GbE PCIe 2.0 QSFP. Exadata Storage Server. 1 (NET0) 41

42 InfiniBand Sun QDR InfiniBand PCIe (Host Channel Adapter, HCA) 1 Oracle ILOM (NET MGT) 1 Sun ZFS Storage NET0( 25 ) NET1( 26 ) InfiniBand QDR InfiniBand 1 Oracle ILOM 1 (NET0) Oracle ILOM. SPARC SuperCluster T4-4 Cisco Catalyst IP. IP NTP VLAN(Virtual Local Area Network),. SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. SPARC SuperCluster T SPARC SuperCluster T4-4.,, Oracle ILOM.. 42 SPARC SuperClusterT

43 PDU. SPARC T4-4 Exadata Storage Server. 1GbE Oracle ILOM Oracle Integrated Lights Out Manager. SPARC T GbE (NET 0 - NET3). NET IPMP. 29 SPARC SuperCluster T4-4. Sun ZFS Storage : 1GbE Oracle ILOM NET0. 2: Oracle ILOM NET0 1GbE NET1. SPARC SuperCluster T4-4 Oracle ILOM 1GbE. SPARC T4-4 1GbE.. 10GbE SPARC T4-4. SCAN(Single Client Access Name) Oracle RAC IP(Virtual IP, VIP). InfiniBand InfiniBand SPARC T4-4, Sun ZFS Storage 7320 Exadata Storage Server. SPARC T4-4 Oracle Oracle RAC Exadata Storage Server Sun ZFS Storage SPARC T4-4 Oracle Solaris Cluster Sun ZFS Storage

44 SPARC SuperClusterT4-4 SPARC SuperCluster T SPARC SuperCluster T4-4 SPARC SuperClusterT4-4 SPARC SuperCluster T SPARC SuperClusterT

45 IP ( ) 1 ( ) SPARC T : : 4-8 : 2-4. (SPARC T4-4 1 IPMP.) IP (SPARC SuperClusterT4-4) SPARC SuperCluster T4-4 IP. 45 IP 45 IP ( ) 50 IP ( ) IP 4 IP. IP SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320 Oracle ILOM IP. IP SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320 IP. InfiniBand IP InfiniBand SPARC T4-4, Exadata Storage Server Sun ZFS Storage SPARC SuperCluster T4-4 InfiniBand SPARC SuperCluster T4-4 Oracle Exadata Exalogic InfiniBand 1 SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 Oracle Exadata Exalogic SPARC T GbE IP 10GbE IP. 5. IP ( ) SPARC SuperCluster T4-4 IP. 46 Oracle ILOM IP ( ) 48 InfiniBand 10GbE IP ( ) 45

46 IP ( ) Oracle ILOM IP ( ) 2 Oracle ILOM IP ( ) ( ) Oracle ILOM Oracle ILOM IP IP PDU-A( ) sscpdua PCU-B( ) sscpdub Exadata Storage Server 6 ssces6-sp cell Exadata Storage Server 5 ssces5-sp cell Exadata Storage Server 4 ssces4-sp cell SPARC T4-4 4 ssccn4-sp ssccn4-m ssccn4-m ssccn4-m ssccn4-m SPARC T4-4 3 ssccn3-sp ssccn3-m ssccn3-m ssccn3-m ssccn3-m Sun ZFS Storage Sun ZFS Storage Sun Datacenter InfiniBand Switch 36( 2) sscsn2-sp sscsn2-m sscsn1-sp sscsn1-m sscnm1-m SPARC SuperClusterT

47 IP ( ) 2 Oracle ILOM IP ( ) ( ) ( ) Oracle ILOM Oracle ILOM IP IP 23 Sun ZFS Storage Sun Disk Shelf Cisco Catalyst Sun Datacenter InfiniBand Switch 36( 1) ssc4948-m sscnm1-m SPARC T4-4 2 ssccn2-sp ssccn2-m ssccn2-m ssccn2-m ssccn2-m SPARC T4-4 1 ssccn1-sp ssccn1-m ssccn1-m ssccn1-m ssccn1-m Exadata Storage Server 3 ssces3-sp cell Exadata Storage Server 2 ssces2-sp cell Exadata Storage Server 1 ssces1-sp cell Sun Datacenter InfiniBand Switch 36( ) sscnm1-m

48 IP ( ) InfiniBand 10GbE IP ( ) 3 InfiniBand 10GbE IP ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP PDU-A( ) PCU-B( ) 42 Exadata Storage Server 6 ssces6-stor Exadata Storage Server 5 ssces5-stor Exadata Storage Server 4 ssces4-stor SPARC T4-4 4 ssccn4-ib ssccn4-tg8 ssccn4-tg7 35 ssccn4-ib ssccn4-tg6 ssccn4-tg5 34 ssccn4-ib ssccn4-tg4 ssccn4-tg3 33 ssccn4-ib ssccn4-tg2 32 ssccn4-tg1 31 SPARC T4-4 3 ssccn3-ib ssccn3-tg8 ssccn3-tg7 30 ssccn3-ib ssccn3-tg6 ssccn3-tg5 29 ssccn3-ib ssccn3-tg4 ssccn3-tg3 28 ssccn3-ib ssccn3-tg2 27 ssccn3-tg SPARC SuperClusterT

49 IP ( ) 3 InfiniBand 10GbE IP ( ) ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP 26 Sun ZFS Storage Sun ZFS Storage Sun Datacenter InfiniBand Switch 36( 2) 23 Sun ZFS Storage Sun Disk Shelf sscsn1-stor2 sscsn1-stor Cisco Catalyst Sun Datacenter InfiniBand Switch 36( 1) 17 SPARC T4-4 2 ssccn2-ib ssccn2-tg8 ssccn2-tg7 16 ssccn2-ib ssccn2-tg6 ssccn2-tg5 15 ssccn2-ib ssccn2-tg4 ssccn2-tg3 14 ssccn2-ib ssccn2-tg2 13 ssccn2-tg

50 IP ( ) 3 InfiniBand 10GbE IP ( ) ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP 12 SPARC T4-4 1 ssccn1-ib ssccn1-tg8 ssccn1-tg7 11 ssccn1-ib ssccn1-tg6 ssccn1-tg5 10 ssccn1-ib ssccn1-tg4 ssccn1-tg3 9 ssccn1-ib ssccn1-tg ssccn1-tg Exadata Storage Server 3 ssces3-stor Exadata Storage Server 2 ssces2-stor Exadata Storage Server 1 ssces1-stor Sun Datacenter InfiniBand Switch 36( ) IP ( ) SPARC SuperCluster T4-4 IP. 51 Oracle ILOM IP ( ) 53 InfiniBand 10GbE IP ( ) 50 SPARC SuperClusterT

51 IP ( ) Oracle ILOM IP ( ) 4 Oracle ILOM IP ( ) ( ) Oracle ILOM Oracle ILOM IP IP PDU-A( ) sscpdua PCU-B( ) sscpdub U U U U Sun ZFS Storage Sun ZFS Storage Sun Datacenter InfiniBand Switch 36( 2) sscsn2-sp sscsn2-m sscsn1-sp sscsn1-m sscnm1-m

52 IP ( ) 4 Oracle ILOM IP ( ) ( ) ( ) Oracle ILOM Oracle ILOM IP IP 23 Sun ZFS Storage Sun Disk Shelf Cisco Catalyst Sun Datacenter InfiniBand Switch 36( 1) ssc4948-m sscnm1-m SPARC T4-4 2 ssccn2-sp ssccn2-m ssccn2-m ssccn2-m ssccn2-m SPARC T4-4 1 ssccn1-sp ssccn1-m ssccn1-m ssccn1-m ssccn1-m Exadata Storage Server 3 ssces3-sp cell Exadata Storage Server 2 ssces2-sp cell Exadata Storage Server 1 ssces1-sp cell Sun Datacenter InfiniBand Switch 36( ) sscnm1-m SPARC SuperClusterT

53 IP ( ) InfiniBand 10GbE IP ( ) 5 InfiniBand 10GbE IP ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP PDU-A( ) PCU-B( ) 42 4U U U U Sun ZFS Storage Sun ZFS Storage Sun Datacenter InfiniBand Switch 36( 2) sscsn1-stor2 sscsn1-stor

54 IP ( ) 5 InfiniBand 10GbE IP ( ) ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP 23 Sun ZFS Storage Sun Disk Shelf Cisco Catalyst Sun Datacenter InfiniBand Switch 36( 1) 17 SPARC T4-4 2 ssccn2-ib ssccn2-tg8 ssccn2-tg7 16 ssccn2-ib ssccn2-tg6 ssccn2-tg5 15 ssccn2-ib ssccn2-tg4 ssccn2-tg3 14 ssccn2-ib ssccn2-tg2 13 ssccn2-tg1 12 SPARC T4-4 1 ssccn1-ib ssccn1-tg8 ssccn1-tg7 11 ssccn1-ib ssccn1-tg6 ssccn1-tg5 10 ssccn1-ib ssccn1-tg4 ssccn1-tg3 9 ssccn1-ib ssccn1-tg2 8 ssccn1-tg Exadata Storage Server 3 ssces3-stor SPARC SuperClusterT

55 IP ( ) 5 InfiniBand 10GbE IP ( ) ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP 5 Exadata Storage Server 2 ssces2-stor Exadata Storage Server 1 ssces1-stor Sun Datacenter InfiniBand Switch 36( ) 55

56 56

57 ohm..,. ( )48. ( ). 57

58 SPARC SuperCluster T mm mm ( ) 1200mm ( ) 1112mm mm mm 36 ( ) 862kg 1,900 ( ) 594kg 1, mm mm mm SPARC SuperClusterT

59 ,2. 280mm(11 ). 79 ( ).. 6 1: 113mm(4.4). 59

60 2: 374mm(14.72 ). 3: 113mm(4.4). 4: 1120mm(44.1 ). 5: 330mm(13 ). 6: 160mm(6.3 ). 7: 280mm(11 ) PDU PDU 15.81kVA 15.02kW 13.80kVA 13.11kW 8.97kVA 8.52kW 6.9kVA 6.5kW. PDU AC.. 60 SPARC SuperClusterT

61 PDU AC., (Uninterruptible Power Supply, UPS), PDU.. IEC PDU PDU SPARC SuperCluster T Oracle. SPARC SuperCluster T4-4 2 (Power Distribution Unit, PDU). 2 PDU. 6 PDU

62 PDU 6 PDU ( ) SPARC SuperCluster T4-4 PDU kva 22kVA PDU Amp 1ph VAC 110.4A(3 x36.8a) 42 C13 6 C19 3 x 50A, 1ph 36.8A ( ) Hubbell CS8264C PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 8 SPARC SuperCluster T4-4 3 PDU 3 kva 24kVA 6444A ph 200 ~ 240VAC PDU Amp 120A(6 x 20A) 62 SPARC SuperClusterT

63 PDU 8 SPARC SuperCluster T4-4 3 PDU ( ) 3 42 C13 6 C19 2 x 60A, 3ph 34.6A ( ) IEC 309-3P4W-IP67 (60A, 250V, AC, 3ph) PDU Hubbell HBL460R/C9W IEC309 60A 3ph 4 PDU 4 kva 2m(6.6 ) PDU 4m(13 ). 9 SPARC SuperCluster T4-4 PDU kva 22kVA ph 200 ~ 240VAC PDU Amp 96A(3 x 32A) 42 C13 6 C19 3 x 32A, 1ph 32A ( ) IEC 309-2P3W-IP44 (32A, 250V, AC, 3ph) PDU Hubbell HBL332R/C9W IEC309 32A 1ph 3 PDU 6 kva 2m(6.6 ) PDU 4m(13 ). 63

64 10 SPARC SuperCluster T4-4 3 PDU 3 kva 24kVA 6445A PDU Amp 3ph 220/380 ~ 240/415VAC 109A(6 x18.1a) 42 C13 6 C19 2 x25a, 3ph 18A ( ) IEC 309-4P5W-IP44 (32A, 400V, AC, 3ph) PDU Hubbell HBL532R/C9W IEC309 32A 3ph 5 PDU 4 kva 2m(6.6 ) PDU 4m(13 ) ,. 64 SPARC SuperClusterT

65 5 ~ 32 C (41 ~ 89.6 F) -40 ~ 70 C (-40 ~ 158 F) 21 ~ 23 C (70 ~ 47 F). 10 ~ 90%, 93% 45 ~ 50%,. 3048m(10000 ) 12000m(40000 ) 900m 300m ,966 BTU/ (56,937 kj/ ) 30,610 BTU/ (32,295 kj/ ) mm(36 ), 914mm(36 )... 65

66 7 11 ( ) CFM 2,498 2,180 CFM 1,417 1, SPARC SuperClusterT

67 ,. 400CFM SPARC SuperCluster T mm mm mm 62 ( ) kg

68 ( ) 721kg : ,. 68 SPARC SuperClusterT

69 2184mm (86 ) 2040mm (80.32 ) 1220 (48 ) 600mm (23.62 ) 1575mm (62 ) 1200mm (47.24 ) 6 6, ( ) 1134kg (2500 ) 1134kg (2500 )

70 SPARC SuperCluster T IP 71 DNS. ( ) 1Gb 10Gb 1 x 1GB 1 SPARC T4-4 1 IPMP. SPARC T x 10GB 1 SPARC T x 10GB SPARC T x 10GB 1 x 1GB SPARC T x 10GB 1 SPARC T x 10GB SPARC T x 10GB 70 IP 71 DNS 41 IP SPARC SuperCluster T4-4 IP. IP SPARC SuperCluster T DNS SPARC SuperClusterT

71 IP DNS.. GNS(Grid Naming Service) SPARC SuperCluster T4-4 IP DNS(Domain Name System)., SCAN(Single Client Access Name) VIP DNS. IP SCAN. 3 DNS. ( DNS) DNS DNS. SCAN SCAN DNS. Oracle Grid Infrastructure Installation Guide for Linux - SCAN DNS - 70 IP 70 71

72 72

73 SPARC SuperCluster T Oracle Fiber Channel ,, Oracle 57 SPARC SuperCluster T4-4 SPARC SuperCluster T SPARC SuperCluster T SPARC SuperCluster T PDU. 73

74 PDU(PDU A) 6 PDU. 3. Oracle ILOM SPARC SuperCluster T4-4 SPARC T Sun ZFS Storage Sun ZFS Storage Sun Storage 7320 NFS, Sun ZFS Storage 7320 IP ping. 5. Exadata Storage. Oracle ILOM Oracle ILOM Exadata Storage Server. 6. SPARC T SPARC T Sun Datacenter Infiniband Switch SPARC SuperCluster T ( ). 95 Fiber Channel 74 SPARC SuperClusterT

75 Oracle Oracle Oracle.. Important Safety Information for Sun Hardware Systems( ). Sun Rack II Safety and Compliance Guide( ). Sun Rack II Power Distribution Units Users Guide( )

76 8. 76 SPARC SuperClusterT

77 (Customer Information Sheet, CIS) mm. 17mm a... b

78 78 SPARC SuperCluster T ( ) SPARC SuperClusterT SPARC SuperClusterT ( 2 ). 4 SPARC SuperClusterT SPARC SuperClusterT

79 랙을 제자리로 이동 접지 케이블 설치(선택 사항) SPARC SuperCluster T4-4 배전 장치(Power Distribution Unit, PDU)는 해당 전원 코드를 통해 접지됩니다. 추가로 접지하려면 섀시 접지 케이블을 SPARC SuperCluster T4-4에 연결합니다. 추가 접지 지점을 사용하면 전류 누설을 좀 더 효과적으로 방출할 수 있습니다. 79

80 PDU.PDU SPARC SuperClusterT

81 ( ) 81

82 mm..,4. ( ) mm.. Oracle. 82 SPARC SuperClusterT

83 9 1: 113mm(4.4). 2: 374mm(14.72 ). 3: 113mm(4.4). 4: 1120mm(44.1 ). 5: 330mm(13 ). 6: 160mm(6.3 ). 7: 280mm(11 )

84 5. 6 SPARC SuperClusterT SPARC SuperCluster T SPARC SuperCluster T GbE SPARC SuperClusterT

85 AC PDU. PDU. PDU-A. PDU-B. PDU 1 6 ( ). 10 PDU 85

86 1:. 2: SPARC SuperClusterT

87 7 8. PDU SPARC SuperClusterT4-4.. B(PDU B). PDU A. PDU B.. 87

88 PDU B ON( ). PDU B PDU B LED. a. SPARCT4-4. LED LED PS1 PS3 LED PS0 PS2 LED LED SPARC T4-4. b. Sun ZFS Storage LED( ). LED ( 10 ). 88 SPARC SuperClusterT

89 LED( )- ( )- ( )- LED Sun ZFS Storage c. Sun Disk Shelf,. LED. LED ( )- ( )- d. Sun Datacenter InfiniBand Switch 36. LED(PS0 LED) - LED(PS1 LED) - e. Cisco Catalyst PS1 LED PS2 LED 4 A(PDU A). PDU A. 89

90 PDU A ON( ). PDU A LED. a. SPARCT4-4. LED LED PS1 PS3 LED PS0 PS2 LED b. Sun ZFS Storage LED LED - - Sun ZFS Storage 7320 IP ping. NET0 IP 45 IP..IP ping 91 SPARC SuperCluster T SPARC SuperClusterT

91 c. Sun Disk Shelf. LED - - d. Exadata Storage Server. LED - Oracle ILOM ( 3 ). LED e. Sun Datacenter InfiniBand Switch 36. LED(PS0 LED) - LED(PS1 LED) - f. Cisco Catalyst PS1 LED PS2 LED SPARC SuperClusterT4-4 USB. 2 Category 5E 1.. Cisco Catalyst SPARC SuperCluster T USB- USB Cisco. USB- (Sun Network QDR InfiniBand ). SPARC SuperCluster T

92 5. Windows. a. >... b. (TCP/IP).. (TCP/IP). c. IP IP. IP Cisco. SPARC T4-4 NET0 IP Oracle SPARC SuperCluster T4-4 IP. NET0 IP 45 IP. d. IP. e.. Linux. a. root. b. ETH0 SPARC SuperClusterT4-4. # ifconfig -a SPARC SuperCluster T4-4. c. ifconfig. # ifconfig eth netmask up ifconfig IPv eth0. 6 PuTTY telnet ssh. SPARC T4-4 IP IP. 92 SPARC SuperClusterT

93 Cisco Catalyst 4948 SPARC SuperCluster T4-4 NET0 IP. IP 45 IP. Oracle Oracle SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 IP IP. Oracle Oracle SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 IP. 10GbE IP DNS(Domain Name System)., SCAN(Single Client Access Name) VIP DNS. 10GbE SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. PCIE 2, 6, 10 14, SPARC T4-4 Sun Dual 10GbE SFP+ PCIe NIC 4 ( 20 (SPARC T4-4 ) ) 10GbE NIC 10m SFP-QSFP 4 8 SFP-QSFP 10GbE. QSFP 10GbE ( : Sun Network 10GbE Switch 72p) 10GbE ( 4 8 ) SFP-QSFP 10GbE. 10GbE SPARC T4-4 SFP+ 10GbE 93

94 10GbE 10GbE 10Gb 1Gb SPARC SuperCluster T4-4 10GbE. SPARC SuperCluster T4-4 10GbE Sun Dual 10GbE SFP+ PCIe NIC. NIC SPARC SuperCluster T GbE (SPARC T4-4 ) 29 SPARC SuperCluster T GbE SFP. SFP 10 SFP-QSFP. 10GbE NIC SFP. 10GbE. 10GbE. 2 SFP-QSFP QSFP 2 10GbE QSFP ( Sun Network 10GbE Switch 72p). 2 SFP-QSFP SFP+ 16 SPARC T4-4 10GbE NIC SFP+. 10GbE NIC SFP+. 94 SPARC SuperClusterT

95 선택적 Fiber Channel 익스프레스 모듈 사용 그림 11 10GbE 클라이언트 액세스 네트워크에 대한 연결 예 그림 범례 1: QSFP 연결이 있는 10GbE 스위치(Sun Network 10GbE Switch 72p 표시) 2: SFP-QSFP 케이블의 QSFP 커넥터 끝, 10GbE 스위치의 QSFP 포트에 연결 3: SFP-QSFP 케이블의 SFP+ 커넥터 끝, SPARC T4-4 서버의 10GbE NIC의 SFP+ 포트에 연결 선택적 Fiber Channel 익스프레스 모듈 사용 선택적 Fiber Channel 익스프레스 모듈은 기존 저장 부속 시스템에서 SPARC SuperCluster T4-4와 통합된 Exadata Storage Server로의 데이터 마이그레이션을 용이하게 하는 데 사용할 수 있습니다. 선택적 Fiber Channel 익스프레스 모듈은 표준 SPARC SuperCluster T4-4 구성에 포함되어 있지 않으므로 개별적으로 구입해야 합니다. 지원되는 옵션은 다음과 같습니다. 95

96 Fiber Channel ExpressModule SG-XPCIEFCGBE-E8-N Sun StorageTek Dual 8Gb FC Dual 1GbE HBA, RoHS-6, Emulex X- ExpressModule SG-XPCIEFCGBE-Q8-N Sun StorageTek Dual 8Gb FC Dual 1GbE HBA, RoHS-6, Qlogic. X- Fiber Channel 10GbE NIC InfiniBand HCA SPARC T (SPARC T4-4 ). SPARC T4-4. SPARC SuperCluster T SPARC SuperCluster T4-4. Fiber Channel. Oracle Solaris 10 Oracle Solaris 11 Fiber Channel Oracle Database 11gR2. Fiber Channel Oracle Database 11gR2. Oracle Fiber Channel GbE. Oracle.. SPARC T4-4 : 96 SPARC SuperClusterT

97 SPARC SuperClusterT4-4 SPARC SuperCluster T Oracle Solaris Dynamic Intimate Shared Memory Exadata Storage Server 111 SPARC SuperCluster 113 CPU SPARC SuperCluster T

98 Oracle Solaris Cluster. # /usr/cluster/bin/cluster shutdown -g 0 -y 2. Ops Center. # /opt/sunwxvmoc/bin/ecadm ha-stop-no-relocate Exadata Storage Server Exadata Storage Server LDom 7. SPARC T4-4. # init 0 8. Sun ZFS Storage Exadata Storage Server Exadata Storage Server. Exadata. archive/cd_ns/e13877_01/doc/doc.112/e13874/maintenance.htm#cegbhcjg 1. CellCLI > LIST GRIDDISK ATTRIBUTES name WHERE asmdeactivationoutcome!= Yes Oracle ASM Exadata Storage Server. 98 SPARC SuperClusterT

99 Exadata Storage Server Oracle ASM Exadata Storage Server. CellCLI> ALTER GRIDDISK ALL INACTIVE. INACTIVE. CellCLI> LIST GRIDDISK WHERE STATUS!= inactive INACTIVE Exadata Storage Server.. 99 Exadata Storage Server. Exadata Storage Server Exadata Storage Server. Exadata Storage Server. Exadata Storage Server Oracle Clusterware. Exadata Storage Server Oracle ASM. Exadata Storage Server. Exadata Storage Server. # shutdown -h -y now LDom SPARC T4-4. LDom, OS. LDom, ( ).,. ( ). 99

100 1 2 3 I/O,. # ldm stop domainname LDom domainname stopped # ldm unbind-domain domainname I/O,. # ldm stop activedomainname LDom activedomainname stopped # ldm unbind-domain activedomainname. # shutdown -i5 -g0 -y. SPARC SuperCluster T EPO SPARC SuperCluster T4-4. PDU. Oracle Exadata Storage Server, SPARC T4-4, Sun ZFS Storage Sun ZFS Storage SPARCT SPARC SuperClusterT

101 4 Exadata Storage Server. Oracle Solaris 11 Oracle Solaris OS. beadm(1m). SPARC SuperCluster.. SPARC Supercluster. Oracle Solaris Oracle Solaris beadm.., beadm create..,,. beadm mount. pkg (-R). beadm create.,. 101

102 . # beadm create BeName@snapshotNamedescription BeName@snapshotdescription. BeName. snapshotdescription. beadm list -s. beadm create -e. beadm activate. Solaris beadm. SPARC.. localsys% ssh systemname -l root Password: Last login: Wed Nov 13 20:27: from dhcp-vpn-r Oracle Corporation SunOS 5.11 solaris April 2011 root@sup46:~# 2 ZFS beadm. root@sup46:~# beadm list BE Active Mountpoint Space Policy Created solaris NR / 2.17G static :01 Active. N (Now) R (Reboot). 102 SPARC SuperClusterT

103 3 ZFS. beadm create solaris_backup beadm list BE Active Mountpoint Space Policy Created solaris NR / 2.17G static :01 solaris_backup K static :01 4. root@sup46:~# beadm activate solaris_backup root@sup46:~# beadm list BE Active Mountpoint Space Policy Created solaris_backup R G static :01 solaris N / 1.86G static :01 5. root@sup46:~# reboot Connection to systemname closed by remote host. Connection to systemname closed. localsys% ssh systemname -l root Password: Last login: Thu Jul 14 14:37: from dhcp-vpn- Oracle Corporation SunOS 5.11 solaris April 2011 root@sup46:~# beadm list BE Active Mountpoint Space Policy Created solaris_backup NR G static :01 solaris - / 4.12G static :01. root@sup46:~# beadm mount solaris_backup /mnt root@sup46:~# df -k /mnt Filesystem 1024-blocks Used Available Capacity Mounted on rpool1/root/solaris_backup % /mnt root@sup46:~# df -k / Filesystem 1024-blocks Used Available Capacity Mounted on rpool1/root/solaris_backup % / root@sup46:~# ls /mnt bin etc lib opt rpool1 system wwss boot export media pkg sbin tmp 103

104 cdrom home micro platform scde usr dev import mnt proc share var devices java net re shared workspace doe kernel nfs4 root src ws beadm umount solaris beadm activate solaris reboot Connection to systemname closed by remote host. Connection to systemname closed. localsys% ssh systemname -l root Password: Last login: Thu Jul 14 14:37: from dhcp-vpn- Oracle Corporation SunOS 5.11 solaris April beadm list BE Active Mountpoint Space Policy Created solaris_backup G static :19 solaris G static :01 root@sup46:~# beadm destroy solaris_backup Are you sure you want to destroy solaris_backup? This action cannot be undone(y/[n]): y root@sup46:~# beadm list BE Active Mountpoint Space Policy Created solaris NR / 4.12G static :01 root@sup46:~# 104 SPARC SuperClusterT

105 DISM Dynamic Intimate Shared Memory 105 DISM 105 DISM DISM DISM(Dynamic Intimate Shared Memory) ASM SPARC SuperCluster Solaris.ASM SPARC SuperCluster DISM ( ). ASM. Solaris 11 Automatic Memory Management. Solaris 11 DISM DISM. 105 DISM. DISM Oracle DISM Oracle Dynamic SGA Tuning of Oracle Database on Oracle Solaris with DISM. technetwork/articles/systems-hardware-architecture/ using-dynamic-intimate-memory-sparc pdf DISM DISM(Dynamic Intimate Shared Memory) ASM SPARC SuperCluster Solaris. 105 DISM. DBA ASMM(Automatic Shared Memory Management). Solaris DISM. SGA_MAX_SIZE / MEMORY_MAX_TARGET / MEMORY_TARGET, SGA_MAX_SIZE SGA_TARGET SGA., 64G SGA. alter system set SGA_TARGET=64G scope=spfile; alter system set SGA_MAX_SIZE=64G scope=spfile; 105

106 alter system set MEMORY_MAX_TARGET=0 scope=spfile; alter system set MEMORY_TARGET=0 scope=spfile; SPARC SuperCluster T4-4, SPARC T Sun ZFS Storage Sun Datacenter InfiniBand Switch Cisco Catalyst en/us/docs/switches/lan/catalyst4900/4948e/installation/guide/ 4948E_ins.html 18 Exadata Storage Server 106 Exadata Storage Server Oracle VM Server for SPARC Oracle Solaris Cluster Oracle Solaris 11 Oracle Solaris Exadata Storage Server Exadata Storage Server Oracle Exadata Smart Flash Cache Oracle I/O. 18 Exadata Storage Server. SPARC SuperCluster T4-4 Exadata Storage Server Exadata Storage Server. /opt/oracle/cell/doc 106 SPARC SuperClusterT

107 SPARC SuperCluster T4-4 Exadata Storage Server. 107 (Write-Through) 108 Exadata Storage Server 110 Exadata Storage Server Oracle ASM Oracle Exadata Storage Server Software User's Guide. 1 2 (Write-Through) Exadata Storage Server. (Write-Back) (Write-Through). (Write-Through) (Write-Back). Exadata Storage Server (Write-Back) Exadata Storage Server. Oracle Exadata Storage Server ( :1,4, )... CellCLI> ALTER CELL bbulearncycletime=" t02:00:00-08:00". CellCLI> LIST CELL ATTRIBUTES bbulearncycletime Exadata Storage Server.. HDD disk controller battery on disk contoller at adapter 0 is going into a learn cycle. This is a normal maintena 3. # /opt/megaraid/megacli/megacli64 -AdpBbuCmd -GetBbuStatus -a0. BBU status for Adapter: 0 BatteryType: ibbu08 Voltage: 3721 mv Current: 541 ma 107

108 Temperature: 43 C BBU Firmware Status: Charging Status : Charging Voltage : OK Temperature : OK Learn Cycle Requested : No Learn Cycle Active : No Learn Cycle Status : OK Learn Cycle Timeout : No I2c Errors Detected : No Battery Pack Missing : No Battery Replacement required : No Remaining Capacity Low : Yes Periodic Learn Required : No Transparent Learn : No Battery state: GasGuageStatus: Fully Discharged : No Fully Charged : No Discharging : No Initialized : No Remaining Time Alarm : Yes Remaining Capacity Alarm: No Discharge Terminated : No Over Temperature : No Charging Terminated : No Over Charged : No Relative State of Charge: 7 % Charger System State: 1 Charger System Ctrl: 0 Charging current: 541 ma Absolute State of Charge: 0% Max Error: 0 % Exit Code: 0x00 Exadata Storage Server Exadata Storage Server. Exadata Storage Server Exadata Storage Server Oracle ASM. Exadata Storage Server Oracle ASM Exadata Storage Server Exadata Storage Server. Exadata Storage Server. 108 SPARC SuperClusterT

109 CellCLI> LIST GRIDDISK ATTRIBUTES name WHERE asmdeactivationoutcome!= Yes Oracle ASM Exadata Storage Server. Exadata Storage Server Oracle ASM. Exadata Storage Server. CellCLI> ALTER GRIDDISK ALL INACTIVE. Exadata Storage Server INACTIVE. CellCLI> LIST GRIDDISK WHERE STATUS!= inactive INACTIVE Exadata Storage Server..... CellCLI> ALTER GRIDDISK ALL ACTIVE Oracle ASM.. CellCLI> LIST GRIDDISK ATTRIBUTES name, asmmodestatus asmmodestatus ONLINE UNUSED.. DATA_CD_00_dm01cel01 DATA_CD_01_dm01cel01 DATA_CD_02_dm01cel01 DATA_CD_02_dm02cel01 DATA_CD_02_dm03cel01 DATA_CD_02_dm04cel01 DATA_CD_02_dm05cel01 DATA_CD_02_dm06cel01 DATA_CD_02_dm07cel01 DATA_CD_02_dm08cel01 ONLINE SYNCING OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE 109

110 DATA_CD_02_dm09cel01 OFFLINE DATA_CD_02_dm10cel01 OFFLINE DATA_CD_02_dm11cel01 OFFLINE Oracle ASM asmmodestatus=online asmmodestatus=unused. Exadata Storage Server Oracle ASM Exadata Storage Server. Exadata Storage Server.. CellCLI> list griddisk attributes name where asmdeactivationoutcome!= Yes DATA_CD_00_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_01_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_02_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_03_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_04_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_05_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_06_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_07_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_08_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_09_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_10_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_11_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" Exadata Storage Server Oracle ASM Oracle ASM. ALTER DISKGROUP diskgroup-name DROP DISK asm-disk-name Oracle ASM. Exadata Storage Server cellip.ora IP. Exadata Storage Server,. DROP CELLDISK celldisk-on-this-lun FORCE Exadata Storage Server Exadata Storage Server. 110 SPARC SuperClusterT

111 SPARC SuperCluster ssctuner Oracle Solaris 10/11 Perl/Korn. SMF. /etc/system ndd parameters /kernel/drv/sd.conf /kernel/drv/ibd.conf DISM NFS. 2. ssctuner, ssctuner. 2 zpool. ssctuner ssctuner ssctuner 112 ssctuner 113 ssctuner Oracle Solaris OS SMF Oracle Solaris System Administration Guide: Common System Management Tasks. cd/e23824_01/html/ /hbrunlevels html#scrolltoc ssctuner ssctuner. # svcs ssctuner 111

112 , ssctuner syslog ssctuner. ssctuner. ssctuner. # svcs -x ssctuner shows svc:/site/application/sysadmin/ssctuner:default (ssctuner for SPARC SuperCluster) State: online since September 28, :30:15 AM PDT See: ssctuner(l) See: /var/svc/log/site-application-sysadmin-ssctuner:default.log Impact: None. # more /var/svc/log/site-application-sysadmin-ssctuner\:default.log [ Sep 28 07:30:00 Disabled. ] [ Sep 28 07:30:00 Rereading configuration. ] [ Sep 28 07:30:10 Enabled. ] [ Sep 28 07:30:10 Executing start method ("/opt/oracle.supercluster/ssctuner.ksh start"). ] ssctuner local0.notice success: Saved rollback for : /etc/system ssctuner local0.notice success: Saved ndd rollback. ssctuner local0.notice success: Saved rollback for : /kernel/drv/sd.conf ssctuner local0.notice success: enabled, version 0.99e. daemon PID= [ Sep 28 07:30:15 Method "start" exited with status 0. ] ssctuner local0.notice success: daemon executing ssctuner local0.notice success: Changes made to /etc/system ssctuner local0.notice success: Changes made to /kernel/drv/sd.conf /var/adm ssctuner. # grep -i ssctuner /var/adm/messages Sep 28 07:30:10 etc6cn04 ssctuner: [ID local0.notice] success: Saved rollback for : /etc/system Sep 28 07:30:10 etc6cn04 ssctuner: [ID local0.notice] success: Saved ndd rollback. Sep 28 07:30:10 etc6cn04 ssctuner: [ID local0.notice] success: Saved rollback for : /kernel/drv/sd.conf Sep 28 07:30:15 etc6cn04 ssctuner: [ID local0.notice] success: enabled, version 0.99e. daemon PID= Sep 28 07:30:15 etc6cn04 ssctuner: [ID local0.notice] success: daemon executing Sep 28 07:30:15 etc6cn04 ssctuner: [ID local0.notice] success: Changes made to /etc/system Sep 28 07:30:15 etc6cn04 ssctuner: [ID local0.notice] success: Changes made to /kernel/drv/sd.conf 113 ssctuner 112 ssctuner ssctuner ndd ssctuner. ssctuner. ssctuner 113 ssctuner.,. 112 SPARC SuperClusterT

113 CPU ssctuner. # svcadm disable ssctuner ssctuner ndd. 113 ssctuner 111 ssctuner 1 2 ssctuner ssctuner. SMF. /opt/oracle.supercluster/ssctuner/etc/ssctuner.conf FALSE. NDD_TUNE=TRUE SYSTEM_TUNE=TRUE SDCONF_TUNE=TRUE ZPOOL_FIX=TRUE NFS_CHECK=TRUE DISM_CHECK=TRUE SMF. 111 ssctuner 112 ssctuner CPU 114 CPU/Memory 114 CPU/ 115 CPU/ 120 CPU/ 113

114 CPU/Memory CPU/Memory SPARC SuperCluster T4-4 CPU. CPU/Memory (setcoremem). CPU ( ). 4 32GB, TB.... CPU/Memory. Oracle Solaris CPU/ 114 CPU/ 120 CPU/ CPU/ CPU,. CPU/Memory CPU/, (%) 0 1, 32Gb 2, % 75% 2, % 50% 2, % 25% 2, SPARC SuperClusterT

115 CPU/ 13 ( ) (%) 4 25% 25% 50% 2, 1 1, % 50% 25% 2, 1 1, % 25% 25% 2, 1 1, % 25% 25% 25% % CPU/ 120 CPU/ 114 CPU/Memory CPU/ CPU root. ldm bind.. setcoremem CPU. setcoremem 2. setcoremem -s 0 CPU. setcoremem -s

116 CPU/ setcoremem -s 0 ( 3) setcoremem -s 1 ( 5).. $ ssh root@etc8cn06-app1 Password: Last login: Mon Jul 11 1:03: Oracle Corporation SunOS May 2012 root@etc8cn06-app1:~# cd /opt/oracle.supercluster/bin root@etc8cn06-app1:/opt/oracle.supercluster/bin/setcoremem/lib/bin#./setcoremem -s 0 Info: A log of the activties is being recorded in /opt/oracle.supercluster/setcoremem/log/modlocality.log A previous configuration that has not been fully applied already exist, Do you want to reconfigure again? (y,n default: n) y Info: The current or Next Power On SP-Config is layout7_ml Info: Found 4 Domains including primary DOMAIN CORES Memory(GB) ====== ===== ========== primary s11-sru2a-dump s11-sru2a-dump s11-sru2a-dump With this configuration adjustment tool, existing domains can be re-configured with new CPU-core count and memory size. The minimum number of Cores for a domain is 4. Adjustments are allowed in multiple of 4 cores. The minimum amount of Memory for a domain is 32 GB, Adjustments are allowed in multiple of 32 GB. Info: Parking cores (for license purposes) ==================================== Parked cores are cores that are not assigned to any domain and are therefore not counted for license purposes. Parked cores are unusable and essentially wasted. A better alternative in most situations is to avoid unwanted license charges by assigning cores to a different domain. This resource adjustment tool cannot be used to modify the number of domains. Do you want to reconfigure cpu and memory for all 4 domains? (y,n default: n) y DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/0 928/0 s11-sru2a-dump-1 4/0 32/0 s11-sru2a-dump-2 4/0 32/0 s11-sru2a-dump-3 4/0 32/0 CPU. 0. Enter 0 as the value for Core or Memory if you wish to abort Valid Core Options are 0, 4, 8, 12, 16, SPARC SuperClusterT

117 CPU/ For [primary ] Enter the number of Cores between 4-20: 4 Valid Memory Options are 0, 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800, 832, 864, 896, 928 For [primary ] Enter Memory size in GB between : 32 DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/4 928/32 s11-sru2a-dump-1 4/0 32/0 s11-sru2a-dump-2 4/0 32/0 s11-sru2a-dump-3 4/0 32/0 CPU/Memory 0. Enter 0 as the value for Core or Memory if you wish to abort Valid Core Options are 0, 4, 8, 12, 16, 20 For [s11-sru2a-dump-1 ] Enter the number of Cores between 4-20: 8 Valid Memory Options are 0, 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800, 832, 864, 896, 928 For [s11-sru2a-dump-1 ] Enter Memory size in GB between : 288 DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/4 928/32 s11-sru2a-dump-1 4/8 32/288 s11-sru2a-dump-2 4/0 32/0 s11-sru2a-dump-3 4/0 32/0 Enter 0 as the value for Core or Memory if you wish to abort Valid Core Options are 0, 4, 8, 12, 16 For [s11-sru2a-dump-2 ] Enter the number of Cores between 4-16: 8 Valid Memory Options are 0, 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672 For [s11-sru2a-dump-2 ] Enter Memory size in GB between : 288 DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/4 928/32 s11-sru2a-dump-1 4/8 32/288 s11-sru2a-dump-2 4/8 32/288 s11-sru2a-dump-3 4/0 32/0 CPU. Enter 0 as the value for Core or Memory if you wish to abort Valid Core Options are 0, 4, 8, 12, 16 For [s11-sru2a-dump-3 ] Enter the number of Cores between 4-16: 8 Valid Memory Options are 0, 32, 64, 96, 128, 160, 192, 224, 256, 288, 117

118 CPU/ 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672 For [s11-sru2a-dump-3 ] Enter Memory size in GB between : 288 DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/4 928/32 s11-sru2a-dump-1 4/8 32/288 s11-sru2a-dump-2 4/8 32/288 s11-sru2a-dump-3 4/0 32/288 [4 Cores] and [128 GB] of Memory will be parked Do you wish to continue with these changes? (y,n default: n) y Please Execute./setcoremem -s 1 and follow the instructions there Execution of above command will cause all domains in this system to be halted and will require a reboot of this domain. Please enusre that you have stopped all applications in all domains on this system, before proceeding with the above command root@etc8cn06-app1:/opt/oracle.supercluster/bin#,. 4 5, ( ). setcoremem -s 1. ( ). root@etc8cn06-app1:/opt/oracle.supercluster/bin#./setcoremem -s 1 -v Info: A log of the activties is being recorded in /opt/oracle.supercluster/log/modlocality.log Info: Executing step 1 - Please do not interrupt until this step is complete The system has 1024 GB memory Info: The current ldoms config is layout7_ml Info: It will remain preserved, and a new config will be added at end of step 2 Info: The name of the new config will be layout7_ml LDom s11-sru2a-dump-3 stopped Warning: Resulting memory size is zero. Domain s11-sru2a-dump-3 uses physically bound core resources Warning: setting required number of cores to zero LDom s11-sru2a-dump-2 stopped Warning: Resulting memory size is zero. Domain s11-sru2a-dump-2 uses physically bound core resources Warning: setting required number of cores to zero LDom s11-sru2a-dump-1 stopped Warning: Resulting memory size is zero. Domain s11-sru2a-dump-1 uses physically bound core resources Warning: setting required number of cores to zero Initiating a delayed reconfiguration operation on the primary domain. 118 SPARC SuperClusterT

119 CPU/ All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots Info: Adding 4 Cores to Primary Domain Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots Info: Done with changes Info: Done with step 1 - system reboot is required Info: After reboot, Please run setcoremem -s 2 root@etc8cn06-app1:/opt/oracle.supercluster/bin# 6 7. setcoremem -s 2. $ ssh root@etc8cn04-app1 Password: Last login: Mon Aug 23 10:44: Oracle Corporation SunOS May 2012 You have new mail. root@ssc6cn06:~# cd /opt/oracle.supercluster/bin root@etc8cn06-app1:/opt/oracle.supercluster/bin#./setcoremem -s 2 -v Info: A log of the activties is being recorded in /opt/oracle.supercluster/log/modlocality.log Info: Executing step 2 - Please do not interrupt until this step is complete Info: Setting CORE Constraints for Domain s11-sru2a-dump-1 Info: Setting COREs to 1,2,9,10,17,18,25,26 for Domain s11-sru2a-dump-1 Info: Setting Memory for Domain s11-sru2a-dump-1 Info: Binding Domain s11-sru2a-dump-1 Info: Starting Domain s11-sru2a-dump-1 LDom s11-sru2a-dump-1 started Info: Setting CORE Constraints for Domain s11-sru2a-dump-2 Info: Setting COREs to 3,4,11,12,19,20,27,28 for Domain s11-sru2a-dump-2 Info: Setting Memory for Domain s11-sru2a-dump-2 Info: Binding Domain s11-sru2a-dump-2 Cannot validate NIC net9 on service domain s11-sru2a-dump-2 Cannot validate path /dev/rdsk/c0t5000c500436febb7d0s1 on service domain s11-sru2a-dump-1 Info: Starting Domain s11-sru2a-dump-2 LDom s11-sru2a-dump-2 started Info: Setting CORE Constraints for Domain s11-sru2a-dump-3 Info: Setting COREs to 5,6,13,14,21,22,29,30 for Domain s11-sru2a-dump-3 119

120 CPU/ Info: Setting Memory for Domain s11-sru2a-dump-3 LDom s11-sru2a-dump-3 started Info: The new SP Config is layout7_ml Info: Done with changes Info: CPU and Memory configuration successfully changed 120 CPU/ 114 CPU/ 114 CPU/Memory CPU/ CPU/... CPU/. SPARC SuperCluster T /opt/oracle.supercluster/setcoremem/log $ ssh root@hostname-app1 Password: Last login: Mon Jul 11 1:53: Oracle Corporation SunOS May 2012 # ldm list-confg factory-default layout6 layout6_ml layout6_ml layout6_ml layout6_ml [current] layout6..ml Modified Layout( ) (MMDDYYYYHHMISS). layout6_ml # ldm set-config layout6 120 SPARC SuperClusterT

121 CPU/ # #. -> cd /SP -> stop /SYS Are you sure you want to stop /SYS (y/n)? y Stopping /SYS -> start /SYS Are you sure you want to start /SYS (y/n)? y Starting /SYS CPU/Memory 115 CPU/ 114 CPU/ 121

122 122

123 SPARC SuperClusterT4-4 SPARC SuperCluster T SPARC SuperCluster T OCM SPARC SuperCluster T EM Exadata SPARC SuperCluster SPARC SuperClusterT4-4 Oracle (ASR) SPARC SuperCluster T ASR 124 ASR 126 ASR Manager 128 ASR Manager 129 Exadata Storage Server SNMP 131 Sun ZFS Storage 7320 ASR 133 SPARC T4-4 ASR : Oracle ILOM 135 SPARC T4-4 ASR : Oracle Solaris SPARC SuperCluster T4-4 ASR ASR (ASR) SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 ASR Manager. ASR Manager SPARC SuperCluster T4-4 HTTPS HTTPS. ASR Manager Oracle. Oracle /. 123

124 ASR ASR. Oracle Premier Support Oracle/Sun SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 MOS(My Oracle Support).. ASR. ASR.,.. Documentation ASR Fault Coverage Information. ASR SMTP SNMP.. ASR. Oracle.. SNMP ASR Manager. Oracle. ASR Oracle ( ASR SPARC SuperCluster T4-4 ASR. 125 ASR 125 : ASR Manager 125 : SPARC SuperCluster T SPARC SuperClusterT

125 : SPARC SuperClusterT ASR ASR. MOS(My Oracle Support).. Oracle Premier Support Oracle/Sun SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 ASR Manager.. additional details( ) Hardware and Network Configuration Recommendations( ). ASR. HTTPS.. Oracle ASR Security Whitepaper. SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 : ASR Manager ASR Manager. ASR Manager, 3.6 Solaris Oracle STB(Services Tool Bundle) : SPARC SuperClusterT4-4 ASR Manager. : Oracle Linux 5.3 Oracle Solaris 10 Update 10(10u10) + Java : JRE/JDK : Exadata

126 ASR Manager Exadata Storage Server: Exadata Storage Server DB 1 Sun ZFS Storage 7320 : AK ASR Manager ASR Manager ASR Manager. 125 ASR ASR Manager. 126 OASM(Oracle Automated Service Manager) 126 Solaris STB(Service Tools Bundle) 127 Oracle ASR( ) 127 ASR Manager 1 2 OASM(Oracle Automated Service Manager) ( OASM ). : Oracle Solaris: pkginfo -l SUNWsasm Oracle Linux: rpm -q SUNWsasm OASM. : Oracle Solaris: pkgadd -d SUNWsasm.version-number.pkg Oracle Linux: rpm -i SUNWsasm.version-number.rpm 1 2 Solaris STB(ServiceTools Bundle) ServicesTools Bundle. Download( ). STB (install_stb.sh).. "install" I 126 SPARC SuperClusterT

127 ASR Manager SNEEP Y Y Doc ID My Oracle Support STB ( ) SNEEP. sneep -a. stclient -E. sneep -s serial-number 1 2 Oracle ASR( ) ASR. : Oracle Solaris: pkgadd -d SUNWswasr.version-number.pkg Oracle Linux: rpm -i SUNWswasr.version-number.rpm asr PATH..profile,.cshrc,.kshrc.bashrc. PATH=$PATH:/opt/SUNWswasr/bin export PATH 1 2 ASR Manager ASR Manager, MOS. ASR Manager. # asr register "1" "alternate URL for Managed OPS use only". 1) transport.oracle.com 127

128 ASR Manager 3.. Proxy server name:? <proxy server name> Proxy port number:? <proxy port number> Proxy authentication; if authentication is not required, enter -. Proxy user: <proxy user name> Proxy password: <proxy password> If this is an NTLM type proxy, enter the information below. Otherwise, enter - NTLM Domain: [?] <NTLM domain name> Enter the host the NTLM authentication request is originating from. Usually this is the hostname of the SASM server. NTLM Host: [?] <NTLM host name> NTLM support is not bundled with SASM but may be added now. 1) Download jcifs from 2) Extract contents and locate the jcifs-*.jar file 3) Enter full path to this file below jcifs jar file: [?] <full path of jcifs jar file> Note: These properties are stored in the /var/opt/sunwsasm/configuration/config.ini file. You can update these properties if needed and then restart SASM. 4 MOS(My Oracle Support). ASR.. :. MOS ASR (Service Request, SR) ASR Manager ASR Manager ASR Manager. # asr show_rules_version # asr show_reg_status ASR. # asr test_connection 128 SPARC SuperClusterT

129 ASR Manager Exadata Storage Server SNMP.. Exadata Storage Server. 1 2 Exadata Storage Server celladmin. Exadata Storage Server SNMP. # cellcli -e alter cell snmpsubscriber=(host = ASR-Manager-name-or-IP-address,port=162,community=public,type=asr) ASR-Manager-name-or-IP-address.. host='asr-manager-name-or-ip-address' ASR Manager DNS.DNS IP, /etc/hosts ASR Manager. type=asr ASR Manager SNMP. community=public.. port=162 SNMP.., ASR. 3 Oracle ILOM ( Oracle ILOM ). # asr list_asset. IP_ADDRESS HOST_NAME SERIAL_NUMBER ASR PROTOCOL SOURCE ssc1cel FMM0CA Enabled SNMP ILOM ssc1cel FMM0CA Enabled SNMP ILOM ssc1cel FMM0CA Enabled SNMP ILOM ssc1cel01-ilom 1234FMM0CA Enabled SNMP,HTTP EXADATA-SW ssc1cel02-ilom 1235FMM0CA Enabled SNMP,HTTP EXADATA-SW ssc1cel03-ilom 1236FMM0CA Enabled SNMP,HTTP EXADATA-SW Exadata Storage Server Oracle ILOM 5. Oracle ILOM

130 ASR Manager 4 ASR Manager Exadata Storage Server Oracle ILOM. # asr activate_asset -i ILOM-IP-address # asr activate_asset -h ILOM-hostname Oracle ILOM ASR. 5 ASR Exadata OS. # asr activate_exadata -i host-management-ip-address -h host-management-hostname -l ILOM-IP-address # asr activate_exadata -i host-management-ip-address -h host-management-hostname -n ILOM-hostname 6 ASR Manager Exadata Storage Server. # asr list_asset Oracle ILOM. IP_ADDRESS HOST_NAME SERIAL_NUMBER ASR PROTOCOL SOURCE ssc1cel FMM0CA Enabled SNMP ILOM ssc1cel FMM0CA Enabled SNMP ILOM ssc1cel FMM0CA Enabled SNMP ILOM ssc1cel01-ilom 1234FMM0CA Enabled SNMP,HTTP EXADATA-SW ssc1cel02-ilom 1235FMM0CA Enabled SNMP,HTTP EXADATA-SW ssc1cel03-ilom 1236FMM0CA Enabled SNMP,HTTP EXADATA-SW 7 8 Exadata Storage Server. # cellcli -e list cell attributes snmpsubscriber Exadata Storage Server SNMP. # cellcli -e alter cell validate snmp type=asr MOS SPARC SuperClusterT4-4 Exadata Storage Server. SPARC SuperClusterT4-4 Exadata Storage Server MOS Exadata Storage Server. 139 SPARC SuperCluster T4-4 ASR. 130 SPARC SuperClusterT

131 ASR Manager ASR MOS 5.3+ (Doc ID ). Sun ZFS Storage 7320 ASR SPARC SuperCluster T4-4 Sun ZFS Storage 7320 Sun ZFS Storage Sun ZFS Storage 7320 IP. Username( ) root root Enter. Configuration( ) SERVICES( ) Services( ). 131

132 ASR Manager 4 Phone Home( ). Phone Home( ) Phone Home( ). 5 Use web proxy( ). 132 SPARC SuperClusterT

133 ASR Manager Host:port( : ). Username( ). Password( ) Privacy Statement( ).OK( ) My Oracle Support OK( ). Sun Inventory and Enable Phone Home(Sun Inventory ). APPLY( ). Service Enable / Disable( / ) Enable( ). SPARC SuperClusterT4-4 Sun ZFS Storage SPARC SuperClusterT4-4 Sun ZFS Storage 7320 MOS Sun ZFS Storage SPARC SuperCluster T4-4 ASR. ASR MOS 5.3+ (Doc ID ). SPARCT4-4 ASR : Oracle ILOM.. SPARC T4-4 Oracle ILOM SPARC T SPARCT4-4 Oracle ILOM. 133

134 ASR Manager 2. # show /SP/alertmgmt/rules # show /SP/alertmgmt/rules/rule-number. # show /SP/alertmgmt/rules/1 Properties: type = snmptrap level = minor destination = destination_port = 0 community_or_username = public snmp_version = 2c testrule = (Cannot show property). ASR Manager IP. show /SP/alertmgmt/rules/rule-number.. Properties: type = snmptrap level = disable destination = destination_port = 0 community_or_username = public 134 SPARC SuperClusterT

135 SPARCT4-4 ASR : Oracle Solaris 11 snmp_version = 1 testrule = (Cannot show property) # set /SP/alertmgmt/rules/unused-rule-number type=snmptrap level=minor destination=ip-address-of-asr-manager snmp_version=2c community_or_username=public ASR Manager. SPARCT4-4 Oracle ILOM. asr> activate_asset -i ILOM-IP-address SPARC SuperClusterT4-4 SPARCT4-4 Oracle ILOM. SPARC SuperClusterT4-4 SPARCT4-4 MOS SPARCT SPARC SuperCluster T4-4 ASR. ASR MOS 5.3+ (Doc ID ). SPARCT4-4 ASR : Oracle Solaris 11.. Oracle Solaris 11 ASR Manager HTTP xml ASR Oracle. ASR Manager asr enable_http_receiver. HTTP ASR Manager HTTP 136 ASR Manager HTTPS ( ) 135

136 SPARCT4-4 ASR : Oracle Solaris ASR Manager Oracle Solaris 11 SPARC T4-4 ASR Manager HTTP Oracle Solaris 11 ASR HTTP ASR Manager root ASR Manager.. # asr show_http_receiver HTTP. # asr enable_http_receiver -p port-number port-number HTTP. HTTP asr disable_http_receiver # asr show_http_receiver HTTP. HTTP. ASR Manager HTTPS ( ) HTTPS ASR Manager HTTP HTTPS/SSL. Jetty HTTPs/SSL. 1 SSL /var/opt/sunwsasm/configuration/jetty/jetty.xml <Call name="addconnector"> SSL. <Call name= addconnector > <Arg> <New class= org.mortbay.jetty.security.sslsocketconnector > 136 SPARC SuperClusterT

137 SPARCT4-4 ASR : Oracle Solaris 11 <Set name= Port >443</Set> <Set name= maxidletime >30000</Set> <Set name= keystore >path-to-keystore</set> <Set name= password >password</set> <Set name= keypassword key-password</set> <Set name= truststore >path-to-keystore</set> <Set name= trustpassword trust-password</set> </New> </Arg> </Call>. java -classpath lib/jetty jar:lib/jetty-util jar org.mortbay.jetty.security.password plaintext-password OBF:(OBF: ) jetty.xml. 2 OASM. Oracle Solaris. # svcadm restart sasm Oracle Linux. # /opt/sunwsasm/bin/sasm stop-instance # /opt/sunwsasm/bin/sasm start-instance 3 URL SSL. ASR Manager Oracle Solaris 11 SPARCT4-4 ASR Manager Oracle Solaris 11 SPARC T SPARCT4-4 root. asr-notify. # svcs asr-notify : svcs: Pattern asr-notify doesn t match any instances asr-notify. # pkg list asr-notify 137

138 SPARCT4-4 ASR : Oracle Solaris 11 : pkg list: no packages matching asr-modify installed asr-notify. # pkg install system/fault-management/asr-notify svcs asr-notify asr-notify. : # svcs asr-notify STATE STIME FMRI online 16:06:05 svc:/system/fm/asr-notify:default asr-notify. 3 ASR Manager. # asradm register -e # asradm register -e Oracle Support. Oracle Support. Enter Orcle SSO User Name: Enter password: Registration complete # asradm list. PROPERTY VALUE Status Successfully Registered with ASR manager System Id system-identification-number Asset Id asset-identification-number User username Endpoint URL ASR Manager. SPARC SuperClusterT4-4 Oracle Solaris 11 SPARCT SPARC SuperClusterT

139 SPARCT4-4 ASR : Oracle Solaris 11 6 SPARC SuperClusterT4-4 SPARCT4-4 MOS SPARCT SPARC SuperClusterT4-4 ASR. ASR MOS 5.3+ (Doc ID ) SPARC SuperClusterT4-4 ASR ASR Manager SPARC SuperClusterT4-4. list_asset SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320 SPARC SuperCluster T4-4 ASR. My Oracle Support( My Oracle Support Dashboard( ) More( )... Settings( ). Settings( ) Administrative( ) Pending ASR Activations( ASR ). ASR. 139

140 SPARCT4-4 ASR : Oracle Solaris SPARC SuperCluster T4-4. Oracle ILOM ASR. ASR. "Problem Category( )" "My - Auto Service Request (ASR) Installation and Configuration Issues( - (ASR) )" SR. 5..ASR Activation(ASR ). ASR Oracle ASR.( : ASR Manager asr activate_asset ) ASR. Contact Name( ):.. "SR ". Street Address 1( 1):. 140 SPARC SuperClusterT

141 OCM... Country( ):. ZIP/Postal Code( ):. "-". Distribution List( ): ASR... ASR. ASR. 6 ASR "Approve( )". SPARC SuperCluster T4-4 My Oracle Support ASR. 7 ASR. # asradm send test -address@company.com. OCM SPARC SuperClusterT4-4 OCM. 141 OCM 142 SPARC T4-4 Oracle Configuration Manager OCM Oracle Configuration Manager (OCM) Oracle. Oracle.. Oracle Configuration Manager. 141

142 OCM Oracle Oracle Configuration Manager ORACLE_HOME. Oracle Configuration Manager.. Oracle Configuration Manager Oracle.. Oracle Configuration Manager Installation and Administration Guide Oracle Configuration Manager Collection Overview SPARCT4-4 Oracle Configuration Manager root. IP 1GbE.. ssh -l root ssc0101-mgmt SPARC T4-4 IP 1GbE.. ssh -l root ssc01db01 Oracle Configuration Manager. Oracle Configuration Manager. /opt/ocm Oracle Configuration Manager. /usr/lib/ocm 142 SPARC SuperClusterT

143 OCM Oracle Configuration Manager Oracle Configuration Manager. Oracle Configuration Manager ocm-install-dir /ocm-install-dir/ccr/bin emccr. emccr Oracle Configuration Manager SPARC T4-4.. emccr Oracle Configuration Manager SPARC T4-4.. SPARCT4-4 Oracle Configuration Manager. /ocm-install-dir/ccr/bin/configccr Oracle Configuration Manager. / ( ) SPARC T4-4 Oracle Configuration Manager. 7. /ocm-install-dir/ccr/admin/scripts/installccrsql.sh collectconfig -s $SID -r SYS XML /ocm-install-dir/ccr/hosts/sparc-t4-4-server/state/review. XML. XML Oracle ORACLE_HOME/ccr/bin/emCCR collect My Oracle Support. 143

144 OCM 10 Dashboard( ) Customize( ). Targets( ). 144 SPARC SuperClusterT

145 OCM

146 EM Exadata SPARC SuperCluster EM Exadata SPARC SuperCluster SPARC SuperCluster 1.1 Oracle Enterprise Manager Exadata Exadata. Control LDOM 1.1 Oracle SuperCluster. SPARC SuperCluster October 2012 QMU. compmon pkg. pkg info compmon pkg list compmon. compmon. : Exadata exadatadiscoveryprecheck.pl catalog.xml. Discovery Precheck Script My Oracle Support exadatadiscoveryprecheck.pl. Oracle Enterprise Manager Exadata Management Getting Started Guide (docs.oracle.com/cd/e24628_01/doc.121/e27442/title.htm) Enterprise Manager Exadata Storage DB. Exadata Storage DB Oracle Exadata Storage Server Oracle Database Exadata Storage Server System Exadata Storage.. Oracle SuperCluster perfquery 1.5.8, HCA Port Errors (ID ).HCA Enterprise Manager.. perfquery -V 146 SPARC SuperClusterT

147 Exalogic SPARC SuperCluster T4-4 Exalogic. 147 Exalogic 147 Exalogic Dept1_Cluster1 GridLink 156 Dept1_Cluster1 SDP JDBC 158 Exalogic SDP InfiniBand Exalogic Oracle EECS(Exalogic Elastic Cloud Software) /, SPARC SuperCluster T4-4. /., CPU.. WebLogic Server. WebLogic Server. Sockets Direct Protocol InfiniBand, TCP CPU. Exalogic SPARC SuperCluster T4-4 Exalogic. Oracle Exalogic Enterprise Deployment Guide Chapter 3, Network, Storage, and Database Preconfiguration ( toc.htm),,,. 147

148 Exalogic Oracle Exalogic Oracle Exalogic Enterprise Deployment Guide Chapter 5, Configuration Oracle Fusion Middleware ( doc.220/e18479/toc.htm) OracleWebLogic Server. Domainname. Domainname. General( ). Enable Exalogic Optimizations(Exalogic ) Save( )... Enable Exalogic Optimizations(Exalogic ).. MBean I/O. -Dweblogic.ScatteredReadsEnabled=true/false KernelMBean.setScatteredReadsEnabled MBean I/O. -Dweblogic.GatheredWritesEnabled=true/false KernelMBean.setGatheredWritesEnabled MBean. -Dweblogic.replication.enableLazyDeserialization=true/false ClusterMBean.setSessionLazyDeserializationEnabled 148 SPARC SuperClusterT

149 Exalogic. java.io.ioexception: Broken pipe... WebLogic. Coherence*web. Oracle Fusion Middleware Exalogic Enterprise Deployment Guide Chapter 8, Deploying a Sample Web Application to and Oracle WebLogic Cluster ( docs.oracle.com/cd/e18476_01/doc.220/e18479/deploy.htm) dizzyworld.ear. Dept1_Cluster Oracle Fusion Middleware Exalogic Enterprise Deployment Guide Section 5.16 Starting Managed Servers on ComputeNode1 and ComputeNode2 ( E18476_01/doc.220/e18479/create_domain.htm#BABEGAFB), Dept1_Cluster1. ( : WLS1). a. Domain Structure( ) Environment( ) Servers( ). Summary of Servers( ). b. WLS1. Settings forwls1(wls1 ). c. Cluster( ). d. Replication Ports( ). Dept_1_Cluster ( : WLS1). a. OracleWebLogic Server. 149

150 Exalogic b. Lock & Edit( ). c. Environment( ) Servers( ). Summary of Servers( ). d. Servers( ) WLS1. e. Protocols( ), Channels( ). f. New( ). g. ReplicationChannel t3 Next( ). h.. : WLS1 IP. : 7005 i. Next( ) Network Channel Properties( ) Enabled( ) Outbound Enabled( ). j. Finish( ). k. Network Channels( ) WLS1 ReplicationChannel. l. Advanced( ) Enable SDP Protocol(SDP ). m. Save( ). n. Activate Changes( ). Dept1_Cluster1.. Dept1_Cluster1 WLS2 ReplicationChannel t WLS3 ReplicationChannel t SPARC SuperClusterT

151 Exalogic Dept1_Cluster1 WLS4 ReplicationChannel t WLS5 ReplicationChannel t WLS6 ReplicationChannel t WLS7 ReplicationChannel t WLS8 ReplicationChannel t Environment( ) > Clusters( ). Summary of Clusters( ). Dept1_Cluster1( ). Settings for Dept1_Cluster1(Dept1_Cluster1 ). Replication( ). Replication Channel( ) ReplicationChannel. Advanced( ) Enable One Way RMI for Replication( RMI ). Save( ).. startweblogic.sh (base_domain bin ) -Djava.net.preferIPv4Stack=true. a. startweblogic.sh.. ${DOMAIN_HOME}/bin/setDomainEnv.sh $* b.. JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true" c.. 151

152 Dept1_Cluster1 GridLink 12. a. Environment( ) > Servers( ). Summary of Servers( ). b. WLS1 ( : WLS1). Settings forwls1(wls1 ). c. Control( ). Server Status( ) WLS1. Start( ). d. WebLogic. netstat -na. Dept1_Cluster1 GridLink Oracle WebLogic Server Oracle RAC JDBC GridLink. ONS(Oracle Notification Service) Oracle RAC GridLink 154 Dept1_Cluster1 GridLink GridLink GridLink Oracle RAC XA 153 SCAN 153 Oracle Wallet GridLink. 152 SPARC SuperClusterT

153 OracleWallet / Oracle RAC.. Oracle RAC. Oracle RAC Oracle Database JDBC Developer's Guide and Reference Fast Connection Failover. fstconfo.htm. WebLogic Server. CPU,, Oracle RAC FAN(Fast Application Notification), GridLink Oracle RAC. XA XA Oracle RAC Oracle RAC.XA RCLB. Oracle RAC. SCAN SCAN(Oracle Single Client Access Name) WebLogic TNS ONS. Oracle RAC SCAN GridLink. SCAN URL. OracleWallet Oracle Wallet ONS. 153

154 OracleWallet Dept1_Cluster1 GridLink Oracle GridLink.. XA. Dept1_Cluster1. 0. Oracle WebLogic Server Services( ), JDBC Datasources( ). Datasources( ) Datasource Name( ) Connection Pool( ) Initial capacity( ) 0. ONS.ONS onsctl. start GridLink OracleWebLogic Server. Lock & Edit( ). Domain Structure( ) Services( ) Data Sources( ). Summary of Data Sources( ) New( ) GridLink Data Source(GridLink ). Create a New JDBC GridLink Data Source( JDBC GridLink ).. Name( ) ( : gridlink). JNDI ( : jdbc/gridlink). Next( ). Transaction Options( ) Supports Global Transactions( ) Next( ). Enter individual listener information( ) Next( ). 154 SPARC SuperClusterT

155 OracleWallet 8. Service Name( ): Service Name( ) Oracle RAC. Service Name( ) myservice. Oracle RAC. Host Name( ) - DNS IP. Oracle GridLink -,. Port( ) -. Database User Name( ): ( : mydatabase). Password( ): ( : mypassword1). Next( ). Oracle Fusion Middleware Oracle WebLogic Server. JDBC URL.. jdbc:oracle:thin:@(description=(address_list=(address=(protocol=tcp)(host=left)(port= Test GridLink Database Connection(GridLink ) Test All Listeners( ). Oracle WebLogic... Next( ). ONS Client Configuration(ONS ). Fan Enabled(FAN ) Oracle FAN. ONS ONS FAN ONS. SCAN(Single Client Access Name) FAN. Next( ). 155

156 Dept1_Cluster1 SDP JDBC Test ONS client configuration(ons ) Test All ONS Nodes( ONS ). Next( ). Select Targets( ) Dept1_Cluster1 All Servers in the cluster( ). Finish( ). Activate Changes( ). SDP JDBC. 156 Dept1_Cluster1 SDP JDBC. Dept1_Cluster1 SDP JDBC Dept1_Cluster1 SDP JDBC Infiniband 156 JDBC SDP 157 Oracle Solaris netstat SDP Infiniband Oracle Database Net Services Administrator's Guide " Configuring SDP Protocol Support for Infinband Network Communication to the Database Server",JDBC SDP InfiniBand. docs/cd/b28359_01/network.111/b28316/performance.htm#i SDP. 1 JDBC SDP. Oracle Fusion Middleware Exalogic Enterprise Deployment Guide Section 7.6 Configuring Grid Link Data Source for Dept1_Cluster1 ( 156 SPARC SuperClusterT

157 Infiniband E18476_01/doc.220/e18479/optimization.htm#BABHEDI), ComputeNode1 ComputeNode2 JDBC GridLink., JDBC URL. (PORT=1522))(CONNECT_DATA=(SERVICE_NAME=myservice))) 2 3 JDBC URL TCP SDP.. jdbc:oracle:thin:@(description=(address=(protocol=sdp)(host=192.x.x.x) (PORT=1522))(CONNECT_DATA=(SERVICE_NAME=myservice))) startweblogic.sh (base_domain bin ) -Djava.net.preferIPv4Stack=true. a. startweblogic.sh.. ${DOMAIN_HOME}/bin/setDomainEnv.sh $* b.. JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true -Doracle.net.SDP=true" c.. 4. a. Environment( ) > Servers( ). Summary of Servers( ). b. WLS1 ( : WLS1). Settings forwls1(wls1 ). c. Control( ). Server Status( ) WLS1. Start( ). Oracle Solaris netstat SDP SPARC SuperCluster T4-4 Oracle Exalogic Elastic Cloud Software Oracle Solaris 11 netstat SDP. Oracle Solaris

158 Exalogic SDP InfiniBand netstat Oracle Solaris 11 SDP. root. # netstat -f sdp -s l, SDP ( ). SDP sdpactiveopens = sdpcurrestab = 748 sdpprfails = 0 sdprejects = 0 sdpoutsegs = sdpindatabytes = sdpoutdatabytes = SDP sdpactiveopens = 0 sdpcurrestab = 0 sdpprfails = 0 sdprejects = 0 sdpinsegs = sdpoutsegs = sdpindatabytes = sdpoutdatabytes = Exalogic SDP InfiniBand SPARC SuperCluster T4-4 Oracle Exalogic Elastic Cloud Software. InfiniBand SDP. SDP SPARC SuperCluster T InfiniBand SDP InfiniBand SDP Oracle RAC 11g 2,. Infiniband Oracle Exalogic Elastic Cloud Software IP Infiniband. 2. SPARC SuperCluster T /etc/hosts InfiniBand IP. IP.. # Added for Listener over IB 158 SPARC SuperClusterT

159 Exalogic SDP InfiniBand ssc01db01-ibvip.mycompany.com ssc01db01-ibvip ssc01db02-ibvip.mycompany.com ssc01db02-ibvip root InfiniBand. # /u01/app/grid/product/ /bin/srvctl add network -k 2 -S / /bondib0. # /u01/app/grid/product/ /bin/crsctl stat res -t grep net ora.net1.network ora.net2.network -- Output indicating new Network resource # /u01/app/grid/product/ /bin/srvctl config network -k 2 Network exists: 2/ / /bondib0, type static -- Output indicating Network resource on the subnet 2 IP. srvctl add vip -n ssc01db01 -A ssc01db01-ibvip/ /bondib0 -k 2 srvctl add vip -n ssc01db02 -A ssc01db02-ibvip/ /bondib0 -k 2 "oracle" (Grid Infrastructure ), 3 VIP. srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522,/SDP:1522 ( InfiniBand) listener_networks init. tnsnames, $ORACLE_HOME/network/admin tnsnames.ora. TNSNAMES.ORA GRID_HOME. tnsnames.ora. IP. LISTENER_IBREMOTE. DBM_IB. 159

160 Exalogic SDP InfiniBand TNSNAMES. init.ora (LISTENER_NETWORKS) ALTER SYSTEM SET LISTENER_NETWORKS. (DESCRIPTION = DBM = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01-scan)(port = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) )) DBM_IB = (DESCRIPTION = (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01db01-ibvip)(port = 1522)) (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01db02-ibvip)(port = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) )) LISTENER_IBREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01db02-ibvip.mycompany.com)(port = 1522)) )) LISTENER_IBLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01db01-ibvip.mycompany.com)(port = 1522)) (ADDRESS = (PROTOCOL = SDP)(HOST = ssc01db01-ibvip.mycompany.com)(port = 1522)) )) LISTENER_IPLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc0101-vip.mycompany.com)(port = 1521)) )) LISTENER_IPREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01-scan.mycompany.com)(port = 1521)) )) 160 SPARC SuperClusterT

161 Exalogic SDP InfiniBand 7 8 listener_networks init. sysdba. SQLPLUS> alter system set listener_networks= ((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE)), ((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE)) scope=both; 7 LISTENER_IB. srvctl stop listener -l LISTENER_IB srvctl start listener -l LISTENER_IB 161

162 162

163 SPARC SuperCluster T InfiniBand Sun ZFS Storage PDU PDU SPARC SuperCluster T4-4 IIB(InfiniBand) IB.IB 169 InfiniBand.. 1Gb/s Cat 5E Sun Datacenter InfiniBand Switch IB ( U1) IB 1( U18) IB 2( U24) SPARC T SPARC T4-4 1( 8) SPARC T4-4 2( 13) SPARC T4-4 3( 27)( ) SPARC T4-4 4( 32)( ) Sun ZFS Storage Sun ZFS Storage ( 25) Sun ZFS Storage ( 26) Exadata Storage Server - 15 Exadata Storage Server 1( 2) 163

164 Exadata Storage Server 2( 4) Exadata Storage Server 3( 6) Exadata Storage Server 4( 37)( ) Exadata Storage Server 5( 39)( ) Exadata Storage Server 6( 41)( ) - 16 PDU AC Sun Datacenter InfiniBand Switch 36 1:NETMGT0 NET MGT 1 2: InfiniBand 0A-17A( ) 3: InfiniBand 0B-17B( ) 164 SPARC SuperClusterT

165 참조 도면 그림 13 SPARC T4-4 서버 그림 범례 1: NET MGT, 네트워크 관리 포트 2: SER MGT, 직렬 관리 포트 3: SER MGT 포트부터 터미널 장치까지의 RJ-45 케이블 - 어댑터가 필요할 수 있습니다. 4: RJ-45 케이블 - SPARC T4-4 서비스 프로세서를 이더넷에 연결합니다. 5: RJ-45 케이블 - 네트워크 포트 NET 0-3을 이더넷에 연결합니다. 6: 전원 코드 165

166 14 Sun ZFS Storage : 1 2: 2 3: LED 4: 5: 6: NET0,1,2,3 7: USB 0, 1 8: HD15 15 Exadata Storage Server 1: 1 2: 2 3: LED 4: 5: 6: NET0,1,2,3 7: USB 0, 1 8: HD SPARC SuperClusterT

167 16 (Cisco Catalyst 4948 ) 1: 2: 1-16, 10/100/1000BASE-T 3: 17-32, 10/100/1000BASE-T 4: 33-48, 10/100/1000BASE-T 5:CON( ), MGT( ) 6: 45-48,

168 참조 도면 그림 17 PDU 회로 차단기 및 AC 소켓 그림 범례 1: PDU A 2: PDU B 168 SPARC SuperCluster T 년 12월

169 InfiniBand InfiniBand IB(InfiniBand). IB IB IB Gb/s 175. IB 14 IB (U1) U1 1B 1 U18 8B U1 0B 2 U24 8B IB IB 1(U18) 1 U18 0A T4-4 4 U32 PCIE-12 P2 U18 0B T4-4 4 U32 PCIE-9 P2 U18 1A T4-4 4 U32 PCIE-7 P2 U18 1B T4-4 4 U32 PCIE-1 P2 U18 2A U18 2B U18 3A Sun ZFS Storage U26 PCIE-0 P1 U18 3B T4-4 2 U13 PCIE-12 P1 U18 4A T4-4 2 U13 PCIE-9 P1 169

170 InfiniBand 15 -IB 1(U18) ( ) 1 U18 4B T4-4 2 U13 PCIE-7 P1 U18 5A T4-4 2 U13 PCIE-1 P1 U18 5B T4-4 3 U27 PCIE-12 P2 U18 6A T4-4 3 U27 PCIE-9 P2 U18 6B T4-4 3 U27 PCIE-7 P2 U18 7A T4-4 3 U27 PCIE-1 P2 U18 7B U18 8A IB 2 U24 8A U18 8B IB U1 1B U18 9A IB 2 U24 9B U18 9B IB 2 U24 9A U18 10A IB 2 U24 10B U18 10B IB 2 U24 10A U18 11A IB 2 U24 11B U18 11B IB 2 U24 11A U18 12A U18 12B T4-4 1 U8 PCIE-12 P1 U18 13A T4-4 1 U8 PCIE-9 P1 U18 13B T4-4 1 U8 PCIE-7 P1 U18 14A T4-4 1 U8 PCIE-1 P1 U18 14B Sun ZFS Storage U25 PCIE-0 P1 U18 15A Exadata Storage Server 6 U41 PCIE-3 P2 U18 15B Exadata Storage Server 5 U39 PCIE-3 P2 U18 16A Exadata Storage Server 4 U37 PCIE-3 P2 U18 16B Exadata Storage Server 3 U6 PCIE-3 P1 U18 17A Exadata Storage Server 2 U4 PCIE-3 P1 U18 17B Exadata Storage Server 1 U2 PCIE-3 P1 170 SPARC SuperClusterT

171 InfiniBand 16 -IB 1(U18) 1 U18 0A U18 0B U18 1A U18 1B U18 2A U18 2B U18 3A Sun ZFS Storage U26 PCIE-0 P1 U18 3B T4-4 2 U13 PCIE-12 P1 U18 4A T4-4 2 U13 PCIE-9 P1 U18 4B T4-4 2 U13 PCIE-7 P1 U18 5A T4-4 2 U13 PCIE-1 P1 U18 5B U18 6A U18 6B U18 7A U18 7B U18 8A IB 2 U24 8A U18 8B IB U1 1B U18 9A IB 2 U24 9B U18 9B IB 2 U24 9A U18 10A IB 2 U24 10B U18 10B IB 2 U24 10A U18 11A IB 2 U24 11B U18 11B IB 2 U24 11A U18 12A U18 12B T4-4 1 U8 PCIE-12 P1 171

172 InfiniBand 16 -IB 1(U18) ( ) 1 U18 13A T4-4 1 U8 PCIE-9 P1 U18 13B T4-4 1 U8 PCIE-7 P1 U18 14A T4-4 1 U8 PCIE-1 P1 U18 14B Sun ZFS Storage U25 PCIE-0 P1 U18 15A U18 15B U18 16A U18 16B Exadata Storage Server 3 U6 PCIE-3 P1 U18 17A Exadata Storage Server 2 U4 PCIE-3 P1 U18 17B Exadata Storage Server 1 U2 PCIE-3 P1 IB IB 2(U24) 2 U24 0A T4-4 4 U32 PCIE-12 P1 U24 0B T4-4 4 U32 PCIE-9 P1 U24 1A T4-4 4 U32 PCIE-7 P1 U24 1B T4-4 4 U32 PCIE-1 P1 U24 2A U24 2B U24 3A Sun ZFS Storage U26 PCIE-0 P2 U24 3B T4-4 2 U13 PCIE-12 P2 U24 4A T4-4 2 U13 PCIE-9 P2 U24 4B T4-4 2 U13 PCIE-7 P2 U24 5A T4-4 2 U13 PCIE-1 P2 U24 5B T4-4 3 U27 PCIE-12 P1 172 SPARC SuperClusterT

173 InfiniBand 17 -IB 2(U24) ( ) 2 U24 6A T4-4 3 U27 PCIE-9 P1 U24 6B T4-4 3 U27 PCIE-7 P1 U24 7A T4-4 3 U27 PCIE-1 P1 U24 7B U24 8A IB 1 U18 8A U24 8B IB U1 0B U24 9A IB 1 U18 9B U24 9B IB 1 U18 9A U24 10A IB 1 U18 10B U24 10B IB 1 U18 10A U24 11A IB 1 U18 11B U24 11B IB 1 U18 11A U24 12A U24 12B T4-4 1 U8 PCIE-12 P2 U24 13A T4-4 1 U8 PCIE-9 P2 U24 13B T4-4 1 U8 PCIE-7 P2 U24 14A T4-4 1 U8 PCIE-1 P2 U24 14B Sun ZFS Storage U25 PCIE-0 P2 U24 15A Exadata Storage Server 6 U41 PCIE 3 P1 U24 15B Exadata Storage Server 5 U39 PCIE 3 P1 U24 16A Exadata Storage Server 4 U37 PCIE 3 P1 U24 16B Exadata Storage Server 3 U6 PCIE-3 P2 U24 17A Exadata Storage Server 2 U4 PCIE-3 P2 U24 17B Exadata Storage Server 1 U2 PCIE-3 P2 173

174 InfiniBand 18 -IB 2(U24) 2 U24 0A U24 0B U24 1A U24 1B U24 2A U24 2B U24 3A Sun ZFS Storage U26 PCIE-0 P2 U24 3B T4-4 2 U13 PCIE-12 P2 U24 4A T4-4 2 U13 PCIE-9 P2 U24 4B T4-4 2 U13 PCIE-7 P2 U24 5A T4-4 2 U13 PCIE-1 P2 U24 5B U24 6A U24 6B U24 7A U24 7B U24 8A IB 1 U18 8A U24 8B IB U1 0B U24 9A IB 1 U18 9B U24 9B IB 1 U18 9A U24 10A IB 1 U18 10B U24 10B IB 1 U18 10A U24 11A IB 1 U18 11B U24 11B IB 1 U18 11A U24 12A U24 12B T4-4 1 U8 PCIE-12 P2 174 SPARC SuperClusterT

175 18 -IB 2(U24) ( ) 2 U24 13A T4-4 1 U8 PCIE-9 P2 U24 13B T4-4 1 U8 PCIE-7 P2 U24 14A T4-4 1 U8 PCIE-1 P2 U24 14B Sun ZFS Storage U25 PCIE-0 P2 U24 15A U24 15B U24 16A U24 16B Exadata Storage Server 3 U6 PCIE-3 P2 U24 17A Exadata Storage Server 2 U4 PCIE-3 P2 U24 17B Exadata Storage Server 1 U2 PCIE-3 P2 (Cisco Catalyst , 16) U19 SPARC SuperCluster T4-4. SPARC T4-4, Sun ZFS Storage 7320, Exadata Storage Server PDU , 46, (10BASE-T/100BASE-TX/1000BASE-T ) 6(1000BASE-X ). 1, 2,

176 19 Cat 5E - (U19) T4-4 4 U32 NET 3,3m 6 T4-4 3 U27 NET 3,3m 7 T4-4 4 U32 NET 2,3m 8 T4-4 3 U27 NET 2,3m 9 T4-4 4 U32 NET 1,3m 10 T4-4 3 U27 NET 1,3m 11 T4-4 4 U32 NET 0,3m 12 T4-4 4 U32 NET MGT,3m 13 T4-4 3 U27 NET-0,3m 14 T4-4 3 U27 NET MGT,3m 15 PDU-A PDU-A NET MGT,1m 16 T4-4 2 U13 NET-1,3m 17 T4-4 2 U13 NET-0,3m 18 T4-4 2 U13 NET MGT,3m 19 PDU-B PDU-B NET MGT,1m 20 T4-4 2 U13 NET-2,3m 21 T4-4 1 U8 NET-0,3m 22 T4-4 1 U8 NETMGT,3m 23 Sun ZFS Storage U26 NET-1,3m 24 T4-4 2 U13 NET-3,3m 25 Sun ZFS Storage U26 NET-0,3m 176 SPARC SuperClusterT

177 19 Cat 5E - (U19) ( ) 26 T4-4 1 U8 NET-1,3m 27 Sun ZFS Storage U26 NET-2,3m 28 T4-4 1 U8 NET-2,3m 29 Sun ZFS Storage U25 NET-1,3m 30 T4-4 1 U8 NET-3,3m 31 Sun ZFS Storage U25 NET-0,3m 32 Sun ZFS Storage U25 NET-2,3m 33 Exadata Storage Server 6 U41 NET-0,3m 34 Exadata Storage Server 6 U41 NET MGT,3m 35 Exadata Storage Server 5 U39 NET-0,3m 36 Exadata Storage Server 5 U39 NET MGT,3m 37 Exadata Storage Server 4 U37 NET-0,3m 38 Exadata Storage Server 4 U37 NET MGT,3m 39 Exadata Storage Server 3 U6 NET-0,3m 40 Exadata Storage Server 3 U6 NETMGT,3m 41 Exadata Storage Server 2 U4 NET-0,3m 42 Exadata Storage Server 2 U4 NETMGT,3m 43 Exadata Storage Server 1 U2 NET-0,3m 44 Exadata Storage Server 1 U2 NETMGT,3m 45 Sun Datacenter InfiniBand Switch 36 2 U24 NET-0,3m 46 Sun Datacenter InfiniBand Switch 36 1 U18 NET-0,3m 47 Sun Datacenter Infiniband Switch 36 U1 NET-0,3m 48 ( ),3m 177

178 20 Cat 5E - (U19) PDU-A PDU-A NET MGT,1m 16 T4-4 2 U13 NET-1,3m 17 T4-4 2 U13 NET-0,3m 18 T4-4 2 U13 NET MGT,3m 19 PDU-B PDU-B NET MGT,1m 20 T4-4 2 U13 NET-2,3m 21 T4-4 1 U8 NET-0,3m 22 T4-4 1 U8 NETMGT,3m 23 Sun ZFS Storage U26 NET-1,3m 24 T4-4 2 U13 NET-3,3m 25 Sun ZFS Storage U26 NET-0,3m 178 SPARC SuperClusterT

179 20 Cat 5E - (U19) ( ) 26 T4-4 1 U8 NET-1,3m 27 Sun ZFS Storage U26 NET-2,3m 28 T4-4 1 U8 NET-2,3m 29 Sun ZFS Storage U25 NET-1,3m 30 T4-4 1 U8 NET-3,3m 31 Sun ZFS Storage U25 NET-0,3m 32 Sun ZFS Storage U25 NET-2,3m Exadata Storage Server 3 U6 NET-0,3m 40 Exadata Storage Server 3 U6 NETMGT,3m 41 Exadata Storage Server 2 U4 NET-0,3m 42 Exadata Storage Server 2 U4 NETMGT,3m 43 Exadata Storage Server 1 U2 NET-0,3m 44 Exadata Storage Server 1 U2 NETMGT,3m 45 Sun Datacenter InfiniBand Switch 36 2 U24 NET-0,3m 46 Sun Datacenter InfiniBand Switch 36 1 U18 NET-0,3m 47 Sun Datacenter Infiniband Switch 36 U1 NET-0,3m 48 ( ),3m 179

180 Sun ZFS Storage 7320 Sun ZFS Storage Sun ZFS Storage Sun ZFS Storage ( 25) Sun ZFS Storage ( 26) Sun Disk Shelf( 20) 21 Sun ZFS Storage 7320 SAS U25 PCIE-1 P0 U26 PCIE-1 P2,7 U25 PCIE-1 P1 U26 PCIE-1 P1,7 U25 PCIE-1 P2 U26 PCIE-1 P0,7 U26 PCIE-2 P1 U20 SAS 1,2m U25 PCIE-2 P1 U20 SAS 0,2m U26 PCIE-2 P0 U20 SAS 1,2m U25 PCIE-2 P0 U20 SAS 0,2m PDU AC 22 (Power Distribution Unit, PDU). 2. PDU PDU.. PDU PDU AC PDU PDU-A/PSU-0 - PDB-B/PDU-1 - PDU-A/PSU-2 - PDB-B/PSU-3 - U Exadata Storage Server 6 U Exadata Storage Server SPARC SuperClusterT

181 3 PDU 22 PDU ( ) PDB-B/PDU-1 PDU-A/PSU PDU-A/PSU-2 - PDB-B/PSU-3 - U Exadata Storage Server 4 U SPARC T4-4 4 U SPARC T4-4 3 U Sun ZFS Storage U Sun ZFS Storage U Sun Datacenter InfiniBand Switch 36 2 U Sun Disk Shelf U U Sun Datacenter InfiniBand Switch 36 1 U SPARC T4-4 2 U SPARC T4-4 1 U Exadata Storage Server 3 U Exadata Storage Server 2 U Exadata Storage Server 1 U Sun Datacenter Infiniband Switch 36 3 PDU 3 AC 23 (Power Distribution Unit, PDU). 2. PDU PDU.. PDU PDU AC

182 3 PDU 23 3 PDU PDU-A/PSU-0 - PDB-B/PDU-1 - PDU-A/PSU-2 - PDB-B/PSU-3 - U Exadata Storage Server 6 U Exadata Storage Server 5 U Exadata Storage Server 4 U SPARC T4-4 4 U SPARC T4-4 3 U Sun ZFS Storage U Sun ZFS Storage U Sun Datacenter InfiniBand Switch 36 2 U Sun Disk Shelf U U Sun Datacenter InfiniBand Switch 36 1 U SPARC T4-4 2 U SPARC T4-4 1 U Exadata Storage Server 3 U Exadata Storage Server 2 U Exadata Storage Server 1 U Sun Datacenter Infiniband Switch SPARC SuperClusterT

183 SPARC SuperClusterT4-4 SPARC SuperCluster T4-4 SPARC SuperCluster T Sun Datacenter InfiniBand Switch 36. SPARC SuperCluster T4-4 Sun Datacenter InfiniBand Switch 36. InfiniBand QSFP(Quad Small Form-factor Pluggable)... 1(U1). 18 (U18) 24(U24)

184 7. 8.,. 184 SPARC SuperClusterT

185 , SPARC SuperCluster T4-4. U1 U18 U24 2 (R1-U1). 185

186 24 1 R1-U24 R1-U R1-U18 R1-U18 2 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R1-U1-P5A R1-U24-P9B ~ R1-U1-P6A R1-U24-P10A ~ R2-U1-P7A R1-U24-P10B ~ R2-U1-P8A R1-U24-P11A ~ R2-U1-P9A R1-U24-P11B ~ R2-U1-P10A R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R1-U1-P5B R1-U18-P9B ~ R1-U1-P6B R1-U18-P10A ~ R2-U1-P7B R1-U18-P10B ~ R2-U1-P8B R1-U18-P11A ~ R2-U1-P9B R1-U18-P11B ~ R2-U1-P10B (R2-U1) R2-U24 R2-U24 1 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A R2-U24-P9A ~ R2-U1-P5A R2-U24-P9B ~ R2-U1-P6A R2-U24-P10A ~ R1-U1-P7A R2-U24-P10B ~ R1-U1-P8A R2-U24-P11A ~ R1-U1-P9A R2-U24-P11B ~ R1-U1-P10A 186 SPARC SuperClusterT

187 25 ( ) 2 R2-U18 R2-U18 1 R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B R2-U18-P9A ~ R2-U1-P5B R2-U18-P9B ~ R2-U1-P6B R2-U18-P10A ~ R1-U1-P7B R2-U18-P10B ~ R1-U1-P8B R2-U18-P11A ~ R1-U1-P9B R2-U18-P11B ~ R1-U1-P10B (R1-U1) R1-U24 R1-U24 2 R1-U R1-U18 R1-U18 2 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R1-U1-P5A R1-U24-P9B ~ R2-U1-P6A R1-U24-P10A ~ R2-U1-P7A R1-U24-P10B ~ R2-U1-P8A R1-U24-P11A ~ R3-U1-P9A R1-U24-P11B ~ R3-U1-P10A R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R1-U1-P5B R1-U18-P9B ~ R2-U1-P6B R1-U18-P10A ~ R2-U1-P7B R1-U18-P10B ~ R2-U1-P8B 187

188 26 ( ) R1-U18 3 R1-U18-P11A ~ R3-U1-P9B R1-U18-P11B ~ R3-U1-P10B (R2-U1) R2-U24 R2-U24 1 R2-U R2-U18 R2-U18 1 R2-U18 3 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A R2-U24-P9A ~ R2-U1-P5A R2-U24-P11A ~ R1-U1-P9A R2-U24-P11B ~ R1-U1-P10A R2-U24-P9B ~ R3-U1-P6A R2-U24-P10A ~ R3-U1-P7A R2-U24-P10B ~ R3-U1-P8A R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B R2-U18-P9A ~ R2-U1-P5B R2-U18-P11A ~ R1-U1-P9B R2-U18-P11B ~ R1-U1-P10B R2-U18-P9B ~ R3-U1-P6B R2-U18-P10A ~ R3-U1-P7B R2-U18-P10B ~ R3-U1-P8B (R3-U1). 188 SPARC SuperClusterT

189 28 3 R3-U24 R3-U24 1 R3-U R3-U18 R3-U18 1 R3-U18 2 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A R3-U24-P9A ~ R3-U1-P5A R3-U24-P9B ~ R1-U1-P6A R3-U24-P10A ~ R1-U1-P7A R3-U24-P10B ~ R1-U1-P8A R3-U24-P11A ~ R2-U1-P9A R3-U24-P11B ~ R2-U1-P10A R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B R3-U18-P9A ~ R3-U1-P5B R3-U18-P9B ~ R1-U1-P6B R3-U18-P10A ~ R1-U1-P7B R3-U18-P10B ~ R1-U1-P8B R3-U18-P11A ~ R2-U1-P9B R3-U18-P11B ~ R2-U1-P10B (R1-U1) R1-U24 R1-U24 2 R1-U24 3 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R2-U1-P5A R1-U24-P9B ~ R2-U1-P6A R1-U24-P10A ~ R3-U1-P7A R1-U24-P10B ~ R3-U1-P8A 189

190 29 ( ) R1-U R1-U18 R1-U18 2 R1-U18 3 R1-U18 4 R1-U24-P11A ~ R4-U1-P9A R1-U24-P11B ~ R4-U1-P10A R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R2-U1-P5B R1-U18-P9B ~ R2-U1-P6B R1-U18-P10A ~ R3-U1-P7B R1-U18-P10B ~ R3-U1-P8B R1-U18-P11A ~ R4-U1-P9B R1-U18-P11B ~ R4-U1-P10B (R2-U1) R2-U24 R2-U24 1 R2-U24 3 R2-U R2-U18 R2-U18 1 R2-U18 3 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A R2-U24-P11A ~ R1-U1-P9A R2-U24-P11B ~ R1-U1-P10A R2-U24-P9A ~ R3-U1-P5A R2-U24-P9B ~ R3-U1-P6A R2-U24-P10A ~ R4-U1-P7A R2-U24-P10B ~ R4-U1-P8A R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B R2-U18-P11A ~ R1-U1-P9B R2-U18-P11B ~ R1-U1-P10B R2-U18-P9A ~ R3-U1-P5B R2-U18-P9B ~ R3-U1-P6B 190 SPARC SuperClusterT

191 30 ( ) R2-U18 4 R2-U18-P10A ~ R4-U1-P7B R2-U18-P10B ~ R4-U1-P8B (R3-U1) R3-U24 R3-U24 1 R3-U24 2 R3-U R3-U18 R3-U18 1 R3-U18 2 R3-U18 4 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A R3-U24-P10A ~ R1-U1-P7A R3-U24-P10B ~ R1-U1-P8A R3-U24-P11A ~ R2-U1-P9A R3-U24-P11B ~ R2-U1-P10A R3-U24-P9A ~ R4-U1-P5A R3-U24-P9B ~ R4-U1-P6A R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B R3-U18-P10A ~ R1-U1-P7B R3-U18-P10B ~ R1-U1-P8B R3-U18-P11A ~ R2-U1-P9B R3-U18-P11B ~ R2-U1-P10B R3-U18-P9A ~ R4-U1-P5B R3-U18-P9B ~ R4-U1-P6B (R4-U1). 191

192 32 4 R4-U24 R4-U24 1 R4-U24 2 R4-U R4-U18 R4-U18 1 R4-U18 2 R4-U18 3 R4-U24-P8A ~ R4-U1-P3A R4-U24-P8B ~ R4-U1-P4A R4-U24-P9A ~ R1-U1-P5A R4-U24-P9B ~ R1-U1-P6A R4-U24-P10A ~ R2-U1-P7A R4-U24-P10B ~ R2-U1-P8A R4-U24-P11A ~ R3-U1-P9A R4-U24-P11B ~ R3-U1-P10A R4-U18-P8A ~ R4-U1-P3B R4-U18-P8B ~ R4-U1-P4B R4-U18-P9A ~ R1-U1-P5B R4-U18-P9B ~ R1-U1-P6B R4-U18-P10A ~ R2-U1-P7B R4-U18-P10B ~ R2-U1-P8B R4-U18-P11A ~ R3-U1-P9B R4-U18-P11B ~ R3-U1-P10B (R1-U1) R1 U24 R1 U24 2 R1 U24 3 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R2-U1-P5A R1-U24-P9B ~ R2-U1-P6A R1-U24-P10A ~ R3-U1-P7A R1-U24-P10B ~ R3-U1-P8A SPARC SuperClusterT

193 33 ( ) R1 U24 4 R1-U24-P11A ~ R4-U1-P9A 10 R1 U24 5 R1-U24-P11B ~ R5-U1-P10A 10 1 R1 U18 R1 U18 2 R1 U18 3 R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R2-U1-P5B R1-U18-P9B ~ R2-U1-P6B R1-U18-P10A ~ R3-U1-P7B R1-U18-P10B ~ R3-U1-P8B 3 3 R1 U18 4 R1-U18-P11A ~ R4-U1-P9B 10 R1 U18 5 R1-U18-P11B ~ R5-U1-P10B 10 (R2-U1) R2 U24 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A 3 R2 U24 1 R2-U24-P11B ~ R1-U1-P10A R2 U24 3 R2 U24 4 R2-U24-P9A ~ R3-U1-P5A R2-U24-P9B ~ R3-U1-P6A R2-U24-P10A ~ R4-U1-P7A R2-U24-P10B ~ R4-U1-P8A R2 U24 5 R2-U24-P11A ~ R5-U1-P9A 10 2 R2 U18 R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B 3 R2 U18 1 R2-U18-P11B ~ R1-U1-P10B R2 U18 3 R2-U18-P9A ~ R3-U1-P5B R2-U18-P9B ~ R3-U1-P6B 193

194 34 ( ) R2 U18 4 R2-U18-P10A ~ R4-U1-P7B R2-U18-P10B ~ R4-U1-P8B R2 U18 5 R2-U18-P11A ~ R5-U1-P9B 10 (R3-U1) R3 U24 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A 3 R3 U24 1 R3-U24-P11A ~ R1-U1-P9A R3 U24 2 R3-U24-P11B ~ R2-U1-P10A R3 U24 4 R3 U R3 U18 R3-U24-P9A ~ R4-U1-P5A R3-U24-P9B ~ R4-U1-P6A R3-U24-P10A ~ R5-U1-P7A R3-U24-P10B ~ R5-U1-P8A R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B 3 R3 U18 1 R3-U18-P11A ~ R1-U1-P9B R3 U18 2 R3-U18-P11B ~ R2-U1-P10B R3 U18 4 R3 U18 5 R3-U18-P9A ~ R4-U1-P5B R3-U18-P9B ~ R4-U1-P6B R3-U18-P10A ~ R5-U1-P7B R3-U18-P10B ~ R5-U1-P8B (R4-U1). 194 SPARC SuperClusterT

195 36 4 R4 U24 R4 U24 1 R4-U24-P8A ~ R4-U1-P3A R4-U24-P8B ~ R4-U1-P4A R4-U24-P10A ~ R1-U1-P7A R4-U24-P10B ~ R1-U1-P8A 3 10 R4 U24 2 R4-U24-P11A ~ R2-U1-P9A R4 U24 3 R4-U24-P11B ~ R3-U1-P10A R4 U R4 U18 R4 U18 1 R4-U24-P9A ~ R5-U1-P5A R4-U24-P9B ~ R5-U1-P6A R4-U18-P8A ~ R4-U1-P3B R4-U18-P8B ~ R4-U1-P4B R4-U18-P10A ~ R1-U1-P7B R4-U18-P10B ~ R1-U1-P8B 3 10 R4 U18 2 R4-U18-P11A ~ R2-U1-P9B R4 U18 3 R4-U18-P11B ~ R3-U1-P10B R4 U18 5 R4-U18-P9A ~ R5-U1-P5B R4-U18-P9B ~ R5-U1-P6B (R5-U1) R5 U24 R5 U24 1 R5 U24 2 R5-U24-P8A ~ R5-U1-P3A R5-U24-P8B ~ R5-U1-P4A R5-U24-P9A ~ R1-U1-P5A R5-U24-P9B ~ R1-U1-P6A R5-U24-P10A ~ R2-U1-P7A R5-U24-P10B ~ R2-U1-P8A R5 U24 3 R5-U24-P11A ~ R3-U1-P9A 195

196 37 ( ) R5 U24 4 R5-U24-P11B ~ R4-U1-P10A 5 R5 U18 R5 U18 1 R5 U18 2 R5-U18-P8A ~ R5-U1-P3B R5-U18-P8B ~ R5-U1-P4B R5-U18-P9A ~ R1-U1-P5B R5-U18-P9B ~ R1-U1-P6B R5-U18-P10A ~ R2-U1-P7B R5-U24-P10B ~ R2-U1-P8B R5 U18 3 R5-U18-P11A ~ R3-U1-P9B R5 U18 4 R5-U18-P11B ~ R4-U1-P10B (R1-U1) R1 U24 R1 U24 2 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R2-U1-P5A R1-U24-P9B ~ R2-U1-P6A 3 R1 U24 3 R1-U24-P10A ~ R3-U1-P7A R1 U24 4 R1-U24-P10B ~ R4-U1-P8A 10 R1 U24 5 R1-U24-P11A ~ R5-U1-P9A 10 R1 U24 6 R1-U24-P11B ~ R6-U1-P10A 10 1 R1 U18 R1 U18 2 R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R2-U1-P5B R1-U18-P9B ~ R2-U1-P6B 3 R1 U18 3 R1-U18-P10A ~ R3-U1-P7B 196 SPARC SuperClusterT

197 38 ( ) R1 U18 4 R1-U18-P10B ~ R4-U1-P8B 10 R1 U18 5 R1-U18-P11A ~ R5-U1-P9B 10 R1 U18 6 R1-U18-P11B ~ R6-U1-P10B 10 (R2-U1) R2 U24 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A 3 R2 U24 1 R2-U24-P11B ~ R1-U1-P10A R2 U24 3 R2-U24-P9A ~ R3-U1-P5A R2-U24-P9B ~ R3-U1-P6A R2 U24 4 R2-U24-P10A ~ R4-U1-P7A R2 U24 5 R2-U24-P10B ~ R5-U1-P8A 10 R2 U24 6 R2-U24-P11A ~ R6-U1-P9A 10 2 R2 U18 R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B 3 R2 U18 1 R2-U18-P11B ~ R1-U1-P10B R2 U18 3 R2-U18-P9A ~ R3-U1-P5B R2-U18-P9B ~ R3-U1-P6B R2 U18 4 R2-U18-P10A ~ R4-U1-P7B R2 U18 5 R2-U18-P10B ~ R5-U1-P8B 10 R2 U18 6 R2-U18-P11A ~ R6-U1-P9B 10 (R3-U1). 197

198 40 3 R3 U24 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A 3 R3 U24 1 R3-U24-P11A ~ R1-U1-P9A R3 U24 2 R3-U24-P11B ~ R2-U1-P10A R3 U24 4 R3-U24-P9A ~ R4-U1-P5A R3-U24-P9B ~ R4-U1-P6A R3 U24 5 R3-U24-P10A ~ R5-U1-P7A R3 U24 6 R3-U24-P10B ~ R6-U1-P8A 3 R3 U18 R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B 3 R3 U18 1 R3-U18-P11A ~ R1-U1-P9B R3 U18 2 R3-U18-P11B ~ R2-U1-P10B R3 U18 4 R3-U18-P9A ~ R4-U1-P5B R3-U18-P9B ~ R4-U1-P6B R3 U18 5 R3-U18-P10A ~ R5-U1-P7B R3 U18 6 R3-U18-P10B ~ R6-U1-P8B (R4-U1) R4 U24 R4-U24-P8A ~ R4-U1-P3A R4-U24-P8B ~ R4-U1-P4A 3 R4 U24 1 R4-U24-P10B ~ R1-U1-P8A 10 R4 U24 2 R4-U24-P11A ~ R2-U1-P9A R4 U24 3 R4-U24-P11B ~ R3-U1-P10A R4 U24 5 R4-U24-P9A ~ R5-U1-P5A R4-U24-P9B ~ R5-U1-P6A 198 SPARC SuperClusterT

199 41 ( ) R4 U24 6 R4-U24-P10A ~ R6-U1-P7A 4 R4 U18 R4-U18-P8A ~ R4-U1-P3B R4-U18-P8B ~ R4-U1-P4B 3 R4 U18 1 R4-U18-P10B ~ R1-U1-P8B 10 R4 U18 2 R4-U18-P11A ~ R2-U1-P9B R4 U18 3 R4-U18-P11B ~ R3-U1-P10B R4 U18 5 R4-U18-P9A ~ R5-U1-P5B R4-U18-P9B ~ R5-U1-P6B R4 U18 6 R4-U18-P10A ~ R6-U1-P7B (R5-U1) R5 U24 R5-U24-P8A ~ R5-U1-P3A R5-U24-P8B ~ R5-U1-P4A 3 R5 U24 1 R5-U24-P10A ~ R1-U1-P7A 10 R5 U24 2 R5-U24-P10B ~ R2-U1-P8A 10 R5 U24 3 R5-U24-P11A ~ R3-U1-P9A R5 U24 4 R5-U24-P11B ~ R4-U1-P10A R5 U R5 U18 R5-U24-P9A ~ R6-U1-P5A R5-U24-P9B ~ R6-U1-P6A R5-U18-P8A ~ R5-U1-P3B R5-U18-P8B ~ R5-U1-P4B 3 R5 U18 1 R5-U18-P10A ~ R1-U1-P7B 10 R5 U18 2 R5-U24-P10B ~ R2-U1-P8B 10 R5 U18 3 R5-U18-P11A ~ R3-U1-P9B R5 U18 4 R5-U18-P11B ~ R4-U1-P10B 199

200 42 ( ) R5 U18 6 R5-U18-P9A ~ R6-U1-P5B R5-U18-P9B ~ R6-U1-P6B (R6-U1) R6 U24 R6 U24 1 R6-U24-P8A ~ R6-U1-P3A R6-U24-P8B ~ R6-U1-P4A R6-U24-P9A ~ R1-U1-P5A R6-U24-P9B ~ R1-U1-P6A 3 10 R6 U24 2 R6-U24-P10A ~ R2-U1-P7A 10 R6 U24 3 R6-U24-P10B ~ R3-U1-P8A R6 U24 4 R6-U24-P11A ~ R4-U1-P9A R6 U24 5 R6-U24-P11B ~ R5-U1-P10A 6 R6 U18 R6-U18-P8A ~ R6-U1-P3B R6-U18-P8B ~ R6-U1-P4B 3 R6 U18 2 R6-U24-P10A ~ R2-U1-P7B 10 R6 U18 1 R6-U18-P9A ~ R1-U1-P5B R6-U18-P9B ~ R1-U1-P6B 10 R6 U18 3 R6-U18-P10B ~ R3-U1-P8B R6 U18 4 R6-U18-P11A ~ R4-U1-P9B R6 U18 5 R6-U18-P11B ~ R5-U1-P10B (R1-U1). 200 SPARC SuperClusterT

201 44 1 R1 U24 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A 3 R1 U24 2 R1-U24-P9A ~ R2-U1-P5A R1 U24 3 R1-U24-P9B ~ R3-U1-P6A R1 U24 4 R1-U24-P10A ~ R4-U1-P7A 10 R1 U24 5 R1-U24-P10B ~ R5-U1-P8A 10 R1 U24 6 R1-U24-P11A ~ R6-U1-P9A 10 R1 U24 7 R1-U24-P11B ~ R7-U1-P10A 10 1 R1 U18 R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B 3 R1 U18 2 R1-U18-P9A ~ R2-U1-P5B R1 U18 3 R1-U18-P9B ~ R3-U1-P6B R1 U18 4 R1-U18-P10A ~ R4-U1-P7B 10 R1 U18 5 R1-U18-P10B ~ R5-U1-P8B 10 R1 U18 6 R1-U18-P11A ~ R6-U1-P9B 10 R1 U18 7 R1-U18-P11B ~ R7-U1-P10B 10 (R2-U1) R2 U24 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A 3 R2 U24 1 R2-U24-P11B ~ R1-U1-P10A R2 U24 3 R2-U24-P9A ~ R3-U1-P5A R2 U24 4 R2-U24-P9B ~ R4-U1-P6A R2 U24 5 R2-U24-P10A ~ R5-U1-P7A 10 R2 U24 6 R2-U24-P10B ~ R6-U1-P8A

202 45 ( ) R2 U24 7 R2-U24-P11A ~ R7-U1-P9A 10 2 R2 U18 R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B 3 R2 U18 1 R2-U18-P11B ~ R1-U1-P10B R2 U18 3 R2-U18-P9A ~ R3-U1-P5B R2 U18 4 R2-U18-P9B ~ R4-U1-P6B R2 U18 5 R2-U18-P10A ~ R5-U1-P7B 10 R2 U18 6 R2-U18-P10B ~ R6-U1-P8B 10 R2 U18 7 R2-U18-P11A ~ R7-U1-P9B 10 (R3-U1) R3 U24 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A 3 R3 U24 1 R3-U24-P11A ~ R1-U1-P9A R3 U24 2 R3-U24-P11B ~ R2-U1-P10A R3 U24 4 R3-U24-P9A ~ R4-U1-P5A R3 U24 5 R3-U24-P9B ~ R5-U1-P6A R3 U24 6 R3-U24-P10A ~ R6-U1-P7A 10 R3 U24 7 R3-U24-P10B ~ R7-U1-P8A 10 3 R3 U18 R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B 3 R3 U18 1 R3-U18-P11A ~ R1-U1-P9B R3 U18 2 R3-U18-P11B ~ R2-U1-P10B R3 U18 4 R3-U18-P9A ~ R4-U1-P5B R3 U18 5 R3-U18-P9B ~ R5-U1-P6B 202 SPARC SuperClusterT

203 46 ( ) R3 U18 6 R3-U18-P10A ~ R6-U1-P7B 10 R3 U18 7 R3-U18-P10B ~ R7-U1-P8B 10 (R4-U1) R4 U24 R4-U24-P8A ~ R4-U1-P3A R4-U24-P8B ~ R4-U1-P4A 3 R4 U24 1 R4-U24-P10B ~ R1-U1-P8A 10 R4 U24 2 R4-U24-P11A ~ R2-U1-P9A R4 U24 3 R4-U24-P11B ~ R3-U1-P10A R4 U24 5 R4-U24-P9A ~ R5-U1-P5A R4 U24 6 R4-U24-P9B ~ R6-U1-P6A R4 U24 7 R4-U24-P10A ~ R7-U1-P7A 10 4 R4 U18 R4-U18-P8A ~ R4-U1-P3B R4-U18-P8B ~ R4-U1-P4B 3 R4 U18 1 R4-U18-P10B ~ R1-U1-P8B 10 R4 U18 2 R4-U18-P11A ~ R2-U1-P9B R4 U18 3 R4-U18-P11B ~ R3-U1-P10B R4 U18 5 R4-U18-P9A ~ R5-U1-P5B R4 U18 6 R4-U18-P9B ~ R6-U1-P6B R4 U18 7 R4-U18-P10A ~ R7-U1-P7B 10 (R5-U1). 203

204 48 5 R5 U24 R5-U24-P8A ~ R5-U1-P3A R5-U24-P8B ~ R5-U1-P4A 3 R5 U24 1 R5-U24-P10A ~ R1-U1-P7A 10 R5 U24 2 R5-U24-P10B ~ R2-U1-P8A 10 R5 U24 3 R5-U24-P11A ~ R3-U1-P9A R5 U24 4 R5-U24-P11B ~ R4-U1-P10A R5 U24 6 R5-U24-P9A ~ R6-U1-P5A R5 U24 7 R5-U24-P9B ~ R7-U1-P6A 5 R5 U18 R5-U18-P8A ~ R5-U1-P3B R5-U18-P8B ~ R5-U1-P4B 3 R5 U18 1 R5-U18-P10A ~ R1-U1-P7B 10 R5 U18 2 R5-U24-P10B ~ R2-U1-P8B 10 R5 U18 3 R5-U18-P11A ~ R3-U1-P9B R5 U18 4 R5-U18-P11B ~ R4-U1-P10B R5 U18 6 R5-U18-P9A ~ R6-U1-P5B R5 U18 7 R5-U18-P9B ~ R7-U1-P6B (R6-U1) R6 U24 R6-U24-P8A ~ R6-U1-P3A R6-U24-P8B ~ R6-U1-P4A 3 R6 U24 1 R6-U24-P9B ~ R1-U1-P6A 10 R6 U24 2 R6-U24-P10A ~ R2-U1-P7A 10 R6 U24 3 R6-U24-P10B ~ R3-U1-P8A R6 U24 4 R6-U24-P11A ~ R4-U1-P9A R6 U24 5 R6-U24-P11B ~ R5-U1-P10A 204 SPARC SuperClusterT

205 49 ( ) R6 U24 7 R6-U24-P9A ~ R7-U1-P5A 6 R6 U18 R6-U18-P8A ~ R6-U1-P3B R6-U18-P8B ~ R6-U1-P4B 3 R6 U18 1 R6-U18-P9B ~ R1-U1-P6B 10 R6 U18 2 R6-U24-P10A ~ R2-U1-P7B 10 R6 U18 3 R6-U18-P10B ~ R3-U1-P8B R6 U18 4 R6-U18-P11A ~ R4-U1-P9B R6 U18 5 R6-U18-P11B ~ R5-U1-P10B R6 U18 7 R6-U24-P9A ~ R7-U1-P5B (R7-U1) R7 U24 R7-U24-P8A ~ R7-U1-P3A R7-U24-P8B ~ R7-U1-P4A 3 R7 U24 1 R7-U24-P9A ~ R1-U1-P5A 10 R7 U24 2 R7-U24-P9B ~ R2-U1-P6A 10 R7 U24 3 R7-U24-P10A ~ R3-U1-P7A 10 R7 U24 4 R7-U24-P10B ~ R4-U1-P8A 10 R7 U24 5 R7-U24-P11A ~ R5-U1-P9A R7 U24 6 R7-U24-P11B ~ R6-U1-P10A 7 R7 U18 R7-U18-P8A ~ R7-U1-P3B R7-U18-P8B ~ R7-U1-P4B 3 R7 U18 1 R7-U18-P9A ~ R1-U1-P5B 10 R7 U18 2 R7-U24-P9B ~ R2-U1-P6B 10 R7 U18 3 R7-U18-P10A ~ R3-U1-P7B 10 R7 U18 4 R7-U18-P10B ~ R4-U1-P8B

206 50 ( ) R7 U18 5 R7-U18-P11A ~ R5-U1-P9B R7 U18 6 R7-U24-P11B ~ R6-U1-P10B (R1-U1) R1 U24 R1-U24-P8A ~ R1-U1-P3A 3 R1 U24 2 R1-U24-P8B ~ R2-U1-P4A R1 U24 3 R1-U24-P9A ~ R3-U1-P5A R1 U24 4 R1-U24-P9B ~ R4-U1-P6A 10 R1 U24 5 R1-U24-P10A ~ R5-U1-P7A 10 R1 U24 6 R1-U24-P10B ~ R6-U1-P8A 10 R1 U24 7 R1-U24-P11A ~ R7-U1-P9A 10 R1 U24 8 R1-U24-P11B ~ R8-U1-P10A 10 1 R1 U18 R1-U18-P8A ~ R1-U1-P3B 3 R1 U18 2 R1-U18-P8B ~ R2-U1-P4B R1 U18 3 R1-U18-P9A ~ R3-U1-P5B R1 U18 4 R1-U18-P9B ~ R4-U1-P6B 10 R1 U18 5 R1-U18-P10A ~ R5-U1-P7B 10 R1 U18 6 R1-U18-P10B ~ R6-U1-P8B 10 R1 U18 7 R1-U18-P11A ~ R7-U1-P8B 10 R1 U18 8 R1-U18-P11B ~ R8-U1-P10B 10 (R2-U1). 206 SPARC SuperClusterT

207 52 2 R2 U24 R2-U24-P8A ~ R2-U1-P3A 3 R2 U24 1 R2-U24-P11B ~ R1-U1-P10A R2 U24 3 R2-U24-P8B ~ R3-U1-P4A R2 U24 4 R2-U24-P9A ~ R4-U1-P5A R2 U24 5 R2-U24-P9B ~ R5-U1-P6A 10 R2 U24 6 R2-U24-P10A ~ R6-U1-P7A 10 R2 U24 7 R2-U24-P10B ~ R7-U1-P8A 10 R2 U24 8 R2-U24-P11A ~ R8-U1-P9A 10 2 R2 U18 R2-U18-P8A ~ R2-U1-P3B 3 R2 U18 1 R2-U18-P11B ~ R1-U1-P10B R2 U18 3 R2-U18-P8B ~ R3-U1-P4B R2 U18 4 R2-U18-P9A ~ R4-U1-P5B R2 U18 5 R2-U18-P9B ~ R5-U1-P6B 10 R2 U18 6 R2-U18-P10A ~ R6-U1-P7B 10 R2 U18 7 R2-U18-P10B ~ R7-U1-P8B 10 R2 U18 8 R2-U18-P11A ~ R8-U1-P9B 10 (R3-U1) R3 U24 R3-U24-P8A ~ R3-U1-P3A 3 R3 U24 1 R3-U24-P11A ~ R1-U1-P9A R3 U24 2 R3-U24-P11B ~ R2-U1-P10A R3 U24 4 R3-U24-P8B ~ R4-U1-P4A R3 U24 5 R3-U24-P9A ~ R5-U1-P5A R3 U24 6 R3-U24-P9B ~ R6-U1-P6A 207

208 53 ( ) R3 U24 7 R3-U24-P10A ~ R7-U1-P7A 10 R3 U24 8 R3-U24-P10B ~ R8-U1-P8A 10 3 R3 U18 R3-U18-P8A ~ R3-U1-P3B 3 R3 U18 1 R3-U18-P11A ~ R1-U1-P9B R3 U18 2 R3-U18-P11B ~ R2-U1-P10B R3 U18 4 R3-U18-P8B ~ R4-U1-P4B R3 U18 5 R3-U18-P9A ~ R5-U1-P5B R3 U18 6 R3-U18-P9B ~ R6-U1-P6B R3 U18 7 R3-U18-P10A ~ R7-U1-P7B 10 R3 U18 8 R3-U18-P10B ~ R8-U1-P8B 10 (R4-U1) R4 U24 R4-U24-P8A ~ R4-U1-P3A 3 R4 U24 1 R4-U24-P10B ~ R1-U1-P8A 10 R4 U24 2 R4-U24-P11A ~ R2-U1-P9A R4 U24 3 R4-U24-P11B ~ R3-U1-P10A R4 U24 5 R4-U24-P8B ~ R5-U1-P4A R4 U24 6 R4-U24-P9A ~ R6-U1-P5A R4 U24 7 R4-U24-P9B ~ R7-U1-P6A 10 R4 U24 8 R4-U24-P10A ~ R8-U1-P7A 10 4 R4 U18 R4-U18-P8A ~ R4-U1-P3B 3 R4 U18 1 R4-U18-P10B ~ R1-U1-P8B 10 R4 U18 2 R4-U18-P11A ~ R2-U1-P9B R4 U18 3 R4-U18-P11B ~ R3-U1-P10B R4 U18 5 R4-U18-P8B ~ R5-U1-P4B 208 SPARC SuperClusterT

209 54 ( ) R4 U18 6 R4-U18-P9A ~ R6-U1-P5B R4 U18 7 R4-U18-P9B ~ R7-U1-P6B 10 R4 U18 8 R4-U18-P10A ~ R8-U1-P7B 10 (R5-U1) R5 U24 R5-U24-P8A ~ R5-U1-P3A 3 R5 U24 1 R5-U24-P10A ~ R1-U1-P7A 10 R5 U24 2 R5-U24-P10B ~ R2-U1-P8A 10 R5 U24 3 R5-U24-P11A ~ R3-U1-P9A R5 U24 4 R5-U24-P11B ~ R4-U1-P10A R5 U24 6 R5-U24-P8B ~ R6-U1-P4A R5 U24 7 R5-U24-P9A ~ R7-U1-P5A R5 U24 8 R5-U24-P9B ~ R8-U1-P6A 10 5 R5 U18 R5-U18-P8A ~ R5-U1-P3B 3 R5 U18 1 R5-U18-P10A ~ R1-U1-P7B 10 R5 U18 2 R5-U24-P10B ~ R2-U1-P8B 10 R5 U18 3 R5-U18-P11A ~ R3-U1-P9B R5 U18 4 R5-U18-P11B ~ R4-U1-P10B R5 U18 6 R5-U18-P8B ~ R6-U1-P4B R5 U18 7 R5-U18-P9A ~ R7-U1-P5B R5 U18 8 R5-U18-P9B ~ R8-U1-P6B 10 (R6-U1). 209

210 56 6 R6 U24 R6-U24-P8A ~ R6-U1-P3A 3 R6 U24 1 R6-U24-P9B ~ R1-U1-P6A 10 R6 U24 2 R6-U24-P10A ~ R2-U1-P7A 10 R6 U24 3 R6-U24-P10B ~ R3-U1-P8A R6 U24 4 R6-U24-P11A ~ R4-U1-P9A R6 U24 5 R6-U24-P11B ~ R5-U1-P10A R6 U24 7 R6-U24-P8B ~ R7-U1-P4A R6 U24 8 R6-U24-P9A ~ R8-U1-P5A 6 R6 U18 R6-U18-P8A ~ R6-U1-P3B 3 R6 U18 1 R6-U18-P9B ~ R1-U1-P6B 10 R6 U18 2 R6-U24-P10A ~ R2-U1-P7B 10 R6 U18 3 R6-U18-P10B ~ R3-U1-P8B R6 U18 4 R6-U18-P11A ~ R4-U1-P9B R6 U18 5 R6-U18-P11B ~ R5-U1-P10B R6 U18 7 R6-U24-P8B ~ R7-U1-P4B R6 U18 8 R6-U18-P9A ~ R8-U1-P5B (R7-U1) R7 U24 R7-U24-P8A ~ R7-U1-P3A 3 R7 U24 1 R7-U24-P9A ~ R1-U1-P5A 10 R7 U24 2 R7-U24-P9B ~ R2-U1-P6A 10 R7 U24 3 R7-U24-P10A ~ R3-U1-P7A 10 R7 U24 4 R7-U24-P10B ~ R4-U1-P8A 10 R7 U24 5 R7-U24-P11A ~ R5-U1-P9A 210 SPARC SuperClusterT

211 57 ( ) R7 U24 6 R7-U24-P11B ~ R6-U1-P10A R7 U24 8 R7-U24-P8B ~ R8-U1-P4A 7 R7 U18 R7-U18-P8A ~ R7-U1-P3B 3 R7 U18 1 R7-U18-P9A ~ R1-U1-P5B 10 R7 U18 2 R7-U24-P9B ~ R2-U1-P6B 10 R7 U18 3 R7-U18-P10A ~ R3-U1-P7B 10 R7 U18 4 R7-U18-P10B ~ R4-U1-P8B 10 R7 U18 5 R7-U18-P11A ~ R5-U1-P9B R7 U18 6 R7-U24-P11B ~ R6-U1-P10B R7 U18 8 R7-U24-P8B ~ R8-U1-P4B (R8-U1) R8 U24 R8-U24-P8A ~ R8-U1-P3A 3 R8 U24 1 R8-U24-P8B ~ R1-U1-P4A 10 R8 U24 2 R8-U24-P9A ~ R2-U1-P5A 10 R8 U24 3 R8-U24-P9B ~ R3-U1-P6A 10 R8 U24 4 R8-U24-P10A ~ R4-U1-P7A 10 R8 U24 5 R8-U24-P10B ~ R5-U1-P8A R8 U24 6 R8-U24-P11A ~ R6-U1-P9A R8 U24 7 R8-U24-P11B ~ R7-U1-P10A 8 R8 U18 R8-U18-P8A ~ R8-U1-P3B 3 R8 U18 1 R8-U18-P8B ~ R1-U1-P4B 10 R8 U18 2 R8-U24-P9A ~ R2-U1-P5B 10 R8 U18 3 R8-U18-P9B ~ R3-U1-P6B 10 R8 U18 4 R8-U18-P10A ~ R4-U1-P7B

212 58 ( ) R8 U18 5 R8-U18-P10B ~ R5-U1-P8B R8 U18 6 R8-U24-P11A ~ R6-U1-P9B R8 U18 7 R8-U24-P1B ~ R7-U1-P10B 212 SPARC SuperClusterT

213 SPARC SuperCluster T4-4 Oracle Exadata Storage. 213 Oracle Exadata Storage Oracle Exadata Storage 225 IP 226 ( ) 244 SPARC SuperCluster T4-4 Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage SPARC SuperCluster T4-4.,. Oracle Exadata Storage SPARC SuperCluster T4-4. Exadata Storage Server Oracle ASM(Oracle Automatic Storage Management). Oracle Exadata Storage. Oracle Exadata Storage SPARC SuperCluster T4-4. Oracle Exadata Storage,. Oracle Exadata Storage. 213

214 59 Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage 18 Exadata Storage Server, 600GB 15K RPM SAS 3TB 7.2K RPM SAS 1 3 Sun Datacenter InfiniBand Switch TB, (KVM) 15kVA PDU( 3, )2 48 Cisco Catalyst , WS-C4948-S 1 2TB. 9 Exadata Storage Server, 600GB 15K RPM SAS 3TB 7.2K RPM SAS * 3 Sun Datacenter InfiniBand Switch TB, (KVM) 15kVA PDU( 3, )2 48 Cisco Catalyst , WS-C4948-S 4 Exadata Storage Server, 600GB 15K RPM SAS 3TB 7.2K RPM SAS * 2 Sun Datacenter InfiniBand Switch TB, (KVM) 15kVA PDU( 3, )2 48 Cisco Catalyst , WS-C4948-S Oracle Exadata Storage. SPARC SuperCluster T4-4., SPARC SuperClusterT

215 60 Exadata 1998mm mm mm mm mm mm 36 ( ) 917.6kg 2023 ( ) 578.3kg 127 ( ) 396.8kg PDU 61 Oracle Exadata Storage 12.6kW 6.9kW 3.4kW 12.9kVA 7.1kVA 3.5kVA 8.8kW 4.8kW 2.4kW 9.0kVA 5.0kVA 2.5kVA 215

216 PDU 216 PDU PDU Oracle Exadata Storage. 3 Oracle. Oracle Exadata Storage 2 (Power Distribution Unit, PDU). 2 PDU. 62 PDU Oracle Exadata Storage PDU kva 15kVA 1ph 200 ~ 240VAC PDU Amp 72A(3 x 24A) 42 C13 6 C SPARC SuperClusterT

217 PDU 63 Oracle Exadata Storage PDU ( ) 24A ( ) NEMA 30A/250VAC 2 /3 L6-30P PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 64 Oracle Exadata Storage 3 PDU 3 kva 14.4kVA 3ph 190 ~ 220VAC PDU Amp 69A(3 x 23A) 42 C13 6 C19 2 x 60A, 3ph 40A ( ) IEC A 4 250VAC 3ph IP 67 PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 65 Oracle Exadata Storage PDU kva 15kVA 1ph 217

218 PDU 65 Oracle Exadata Storage PDU ( ) 220 ~ 240VAC PDU Amp 72A(3 x 24A) 42 C C19 25A ( ) IEC A 3 250VAC IP44 PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 66 Oracle Exadata Storage 3 PDU kva 14.4kVA 3ph 220/380 ~ 240/415VAC PDU Amp 62.7A(3 x 20.9 A) 42 C13 6 C19 2 x 25A 25A ( ) IEC A, 5 230/400VAC PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 218 SPARC SuperClusterT

219 ,. 67, 5 ~ 32 C (41 ~ 89.6 F) -40 ~ 70 C (-40 ~ 158 F) 21 ~ 23 C (70 ~ 47 F). 10 ~ 90%, 93% 45 ~ 50%,. 3048m(10000 ) 12000m(40000 ) 900m 300m BTU/ kj/ 43,000 45,400 30,100 31,800 BTU/ kj/ 23,600 24,900 16,500 17,

220 68 ( ) BTU/ kj/ 11,600 12,250 8,100 8, mm(36 ), 914mm(36 ) SPARC SuperClusterT

221 18 69 ( ) CFM 2,000 1,390 CFM 1, CFM

222 . 400CFM Exadata 2159mm mm mm 62 ( ) kg 2207 ( ) 659.6kg 1454 ( ) 475.3kg SPARC SuperClusterT

223 DNS IP Oracle Exadata Storage Exadata Storage Server.. Exadata Storage Server. 1 (NET0) InfiniBand Sun QDR InfiniBand PCIe (Host Channel Adapter, HCA) 1 Oracle ILOM (NET MGT) 1 Oracle Exadata Storage Cisco Catalyst IP. IP NTP Oracle Exadata Storage. Oracle Exadata Storage

224 IP Exadata Storage Server.,, Oracle ILOM.. PDU. Exadata Storage Server 2. 1GbE Oracle ILOM Oracle Integrated Lights Out Manager. InfiniBand InfiniBand Exadata Storage Server. Oracle Exadata Storage IP 71 DNS IP,. Oracle Exadata Storage. 72 Oracle Exadata Storage IP (Exadata Storage Server 1 ) Oracle ILOM 18 (Exadata Storage Server 1 ) 4 (InfiniBand 3 1 ) PDU 2 (PDU ) 224 SPARC SuperClusterT

225 IP 72 Oracle Exadata Storage IP ( ) 24 9 (Exadata Storage Server 1 ) Oracle ILOM 9 (Exadata Storage Server 1 ) 4 (InfiniBand 3 1 ) PDU 2 (PDU ) 13 4 (Exadata Storage Server 1 ) Oracle ILOM 4 (Exadata Storage Server 1 ) 3 (InfiniBand 2, 1 ) PDU 2 (PDU ) 71 DNS IP 226 ( ) Oracle Exadata Storage Oracle Exadata Storage SPARC SuperCluster T GbE Oracle Exadata Storage. IP 73 Oracle Exadata Storage IP NET0 IP Oracle ILOM IP InfiniBand IP Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server

226 ( ) 73 Oracle Exadata Storage IP ( ) NET0 IP Oracle ILOM IP InfiniBand IP Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Exadata Storage Server Sun Datacenter InfiniBand Switch Sun Datacenter InfiniBand Switch Sun Datacenter InfiniBand Switch PDU-A PDU-B ( ) Oracle Exadata Storage. 227 Oracle Exadata Storage 231 Oracle ILOM InfiniBand 226 SPARC SuperClusterT

227 Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage.. 227

228 Oracle Exadata Storage 19 Oracle Exadata Storage Oracle Exadata Storage SPARC SuperClusterT

229 Oracle Exadata Storage 20 Oracle Exadata Storage * U21 * * * Oracle Exadata Storage.. 229

230 Oracle Exadata Storage 21 Oracle Exadata Storage 230 SPARC SuperClusterT

231 Oracle ILOM Oracle ILOM Oracle ILOM. Oracle ILOM NET MGT Oracle Exadata Storage 21. Oracle Exadata Storage Oracle ILOM. 74 Oracle Exadata Storage Oracle ILOM U41 Exadata Storage Server 2 U39 Exadata Storage Server 4 U37 Exadata Storage Server 6 U35 Exadata Storage Server 8 U33 Exadata Storage Server 10 U31 Exadata Storage Server 12 U29 Exadata Storage Server 14 U27 Exadata Storage Server 18 U25 Exadata Storage Server 22 U18 Exadata Storage Server 26 U16 Exadata Storage Server 30 U14 Exadata Storage Server 32 U12 Exadata Storage Server 34 U10 Exadata Storage Server 36 U8 Exadata Storage Server 38 U6 Exadata Storage Server 40 U4 Exadata Storage Server 42 U2 Exadata Storage Server 44 Oracle Exadata Storage Oracle ILOM. 231

232 75 Oracle Exadata Storage Oracle ILOM U18 Exadata Storage Server 26 U16 Exadata Storage Server 30 U14 Exadata Storage Server 32 U12 Exadata Storage Server 34 U10 Exadata Storage Server 36 U8 Exadata Storage Server 38 U6 Exadata Storage Server 40 U4 Exadata Storage Server 42 U2 Exadata Storage Server 44 Oracle Exadata Storage Oracle ILOM. 76 Oracle Exadata Storage Oracle ILOM U8 Exadata Storage Server 38 U6 Exadata Storage Server 40 U4 Exadata Storage Server 42 U2 Exadata Storage Server 44. Net-0 Oracle Exadata Storage 21. Oracle Exadata Storage. 77 Oracle Exadata Storage U41 Exadata Storage Server SPARC SuperClusterT

233 77 Oracle Exadata Storage ( ) U39 Exadata Storage Server 3 U37 Exadata Storage Server 5 U35 Exadata Storage Server 7 U33 Exadata Storage Server 9 U31 Exadata Storage Server 11 U29 Exadata Storage Server 13 U27 Exadata Storage Server 17 U25 Exadata Storage Server 21 U24 Sun Datacenter InfiniBand Switch U20 Sun Datacenter InfiniBand Switch U18 Exadata Storage Server 25 U16 Exadata Storage Server 29 U14 Exadata Storage Server 31 U12 Exadata Storage Server 33 U10 Exadata Storage Server 35 U8 Exadata Storage Server 37 U6 Exadata Storage Server 39 U4 Exadata Storage Server 41 U2 Exadata Storage Server 43 U1 Sun Datacenter InfiniBand Switch PDU-A PDU 15 PDU-B PDU 19 Oracle Exadata Storage. 78 Oracle Exadata Storage U24 Sun Datacenter InfiniBand Switch

234 78 Oracle Exadata Storage ( ) U20 Sun Datacenter InfiniBand Switch U18 Exadata Storage Server 25 U16 Exadata Storage Server 29 U14 Exadata Storage Server 31 U12 Exadata Storage Server 33 U10 Exadata Storage Server 35 U8 Exadata Storage Server 37 U6 Exadata Storage Server 39 U4 Exadata Storage Server 41 U2 Exadata Storage Server 43 U1 Sun Datacenter InfiniBand Switch PDU-A PDU 15 PDU-B PDU 19 Oracle Exadata Storage. 79 Oracle Exadata Storage U24 Sun Datacenter InfiniBand Switch U20 Sun Datacenter InfiniBand Switch U8 Exadata Storage Server 37 U6 Exadata Storage Server 39 U4 Exadata Storage Server 41 U2 Exadata Storage Server 43 PDU-A PDU 15 PDU-B PDU SPARC SuperClusterT

235 (Power Distribution Unit, PDU). PDU-A CMA,4. PDU Oracle Exadata Storage. 80 Oracle Exadata Storage PDU PDU-A/PS-00 PDU-B/PS-01 U41 G5-6 G0-0 2 U39 G5-3 G0-3 2 U37 G5-0 G0-6 2 U35 G4-6 G1-0 2 U33 G4-4 G1-2 2 U31 G4-2 G1-4 2 U29 G3-6 G2-0 2 U27 G3-5 G2-1 2 U25 G3-3 G2-3 2 U24 G3-1 G2-5 2 U23 G3-0 U22 G2-5 G3-1 1 U21 G3-0 G2-6 2 U20 G2-4 G3-2 2 U18 G2-2 G3-4 2 U16 G1-6 G4-0 2 U14 G2-0 G3-6 2 U12 G1-4 G4-2 2 U10 G1-2 G4-4 2 U8 G1-0 G4-6 2 U6 G0-6 G

236 80 Oracle Exadata Storage PDU ( ) PDU-A/PS-00 PDU-B/PS-01 U4 G0-4 G5-2 2 U2 G0-2 G5-4 2 U1 G0-0 G5-6 2 PDU Oracle Exadata Storage. 81 Oracle Exadata Storage PDU PDU-A/PS-00 PDU-B/PS-01 U24 G3-1 G2-5 2 U23 G3-0 U22 G2-5 G3-1 1 U21 G3-0 G2-6 2 U20 G2-4 G3-2 2 U18 G2-2 G3-4 2 U16 G1-6 G4-0 2 U14 G2-0 G3-6 2 U12 G1-4 G4-2 2 U10 G1-2 G4-4 2 U8 G1-0 G4-6 2 U6 G0-6 G5-0 2 U4 G0-4 G5-2 2 U2 G0-2 G5-4 2 PDU Oracle Exadata Storage. 82 Oracle Exadata Storage PDU PDU-A/PS-00 PDU-B/PS-01 U24 G3-1 G SPARC SuperClusterT

237 3 82 Oracle Exadata Storage PDU ( ) PDU-A/PS-00 PDU-B/PS-01 U23 G3-0 U22 G2-5 G3-1 1 U21 G3-0 G2-6 2 U20 G2-4 G3-2 2 U8 G1-0 G4-6 2 U6 G0-6 G5-0 2 U4 G0-4 G5-2 2 U2 G0-2 G (Power Distribution Unit, PDU) 3. PDU-A CMA,4. PDU Oracle Exadata Storage. 83 Oracle Exadata Storage 3 PDU PDU-A/PS-00 PDU-B/PS-01 U41 G5-6 G2-0 2 U39 G5-3 G2-3 2 U37 G5-0 G2-6 2 U35 G4-6 G1-0 2 U33 G4-4 G1-2 2 U31 G4-2 G1-4 2 U29 G3-6 G0-0 2 U27 G3-5 G0-1 2 U25 G3-3 G0-3 2 U24 G3-1 G0-5 2 U23 G

238 3 83 Oracle Exadata Storage 3 PDU ( ) PDU-A/PS-00 PDU-B/PS-01 U22 G2-5 G5-1 1 U21 G3-0 G0-6 2 U20 G2-4 G5-2 2 U18 G2-2 G5-4 2 U16 G1-6 G4-0 2 U14 G2-0 G5-6 2 U12 G1-4 G4-2 2 U10 G1-2 G4-4 2 U8 G1-0 G4-6 2 U6 G0-6 G3-0 2 U4 G0-4 G3-2 2 U2 G0-2 G3-4 2 U1 G0-0 G3-6 2 PDU Oracle Exadata Storage. 84 Oracle Exadata Storage 3 PDU PDU-A/PS-00 PDU-B/PS-01 U24 G3-1 G0-5 2 U23 G5-0 U22 G2-5 G5-1 1 U21 G3-0 G0-6 2 U20 G2-4 G5-2 2 U18 G2-2 G5-4 2 U16 G1-6 G4-0 2 U14 G2-0 G5-6 2 U12 G1-4 G4-2 2 U10 G1-2 G SPARC SuperClusterT

239 InfiniBand 84 Oracle Exadata Storage 3 PDU ( ) PDU-A/PS-00 PDU-B/PS-01 U8 G1-0 G4-6 2 U6 G0-6 G3-0 2 U4 G0-4 G3-2 2 U2 G0-2 G3-4 2 U1 G0-0 G3-6 2 PDU Oracle Exadata Storage. 85 Oracle Exadata Storage 3 PDU PDU-A/PS-00 PDU-B/PS-01 U24 G3-1 G0-5 2 U23 G5-0 U22 G2-5 G5-1 1 U21 G3-0 G0-6 2 U20 G2-4 G5-2 2 U8 G1-0 G4-6 2 U6 G0-6 G3-0 2 U4 G0-4 G3-2 2 U2 G0-2 G3-4 2 InfiniBand InfiniBand. Sun Datacenter InfiniBand Switch 36 1, Oracle Exadata Storage InfiniBand,. 86 Oracle Exadata Storage InfiniBand InfiniBand U24 0A U41 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand 239

240 InfiniBand 86 Oracle Exadata Storage InfiniBand ( ) InfiniBand U24 0B U39 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 1A U37 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 1B U35 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 2A U33 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 2B U31 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 3A U29 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 4A U27 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U24 5A U25 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U24 13A U18 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 14A U16 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 14B U14 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 15A U12 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 15B U10 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 16A U8 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 16B U6 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17A U4 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17B U2 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 0A U41 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 0B U39 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 1A U37 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 1B U35 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 2A U33 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 2B U31 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 3A U29 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 4A U27 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U20 5A U25 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U20 13A U18 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand 240 SPARC SuperClusterT

241 InfiniBand 86 Oracle Exadata Storage InfiniBand ( ) InfiniBand U20 14A U16 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U20 14B U14 Exadata Storage Server PCIe 3, P1 2 QDR InfiniBand U20 15A U12 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 15B U10 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 16A U8 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 16B U6 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17A U4 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17B U2 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 9B U24 Sun Datacenter InfiniBand Switch 36 U20 10B U24 Sun Datacenter InfiniBand Switch 36 U20 11B U24 Sun Datacenter InfiniBand Switch 36 U20 8A U24 Sun Datacenter InfiniBand Switch 36 U20 9A U24 Sun Datacenter InfiniBand Switch 36 U20 10A U24 Sun Datacenter InfiniBand Switch 36 U20 11A U24 Sun Datacenter InfiniBand Switch 36 U1 1B U20 Sun Datacenter InfiniBand Switch 36 U1 0B U24 Sun Datacenter InfiniBand Switch 36 9A 10A 11A 8A 9B 10B 11B 8B 8B 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 3 QDR InfiniBand 3 QDR InfiniBand Oracle Exadata Storage InfiniBand,. 241

242 InfiniBand 87 Oracle Exadata Storage InfiniBand InfiniBand U24 13A U18 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 14A U16 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 14B U14 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 15A U12 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 15B U10 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 16A U8 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 16B U6 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17A U4 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17B U2 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 13A U18 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U20 14A U16 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U20 14B U14 Exadata Storage Server PCIe 3, P1 2 QDR InfiniBand U20 15A U12 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 15B U10 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 16A U8 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 16B U6 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17A U4 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17B U2 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 9B U24 Sun Datacenter InfiniBand Switch 36 U20 10B U24 Sun Datacenter InfiniBand Switch 36 U20 11B U24 Sun Datacenter InfiniBand Switch 36 U20 8A U24 Sun Datacenter InfiniBand Switch 36 U20 9A U24 Sun Datacenter InfiniBand Switch 36 9A 10A 11A 8A 9B 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 242 SPARC SuperClusterT

243 InfiniBand 87 Oracle Exadata Storage InfiniBand ( ) InfiniBand U20 10A U24 Sun Datacenter InfiniBand Switch 36 U20 11A U24 Sun Datacenter InfiniBand Switch 36 U1 1B U20 Sun Datacenter InfiniBand Switch 36 U1 0B U24 Sun Datacenter InfiniBand Switch 36 10B 11B 8B 8B 2 QDR InfiniBand 2 QDR InfiniBand 3 QDR InfiniBand 3 QDR InfiniBand Oracle Exadata Storage InfiniBand,. 88 Oracle Exadata Storage InfiniBand InfiniBand U24 16A U8 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 16B U6 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17A U4 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17B U2 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 16A U8 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U20 16B U6 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17A U4 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17B U2 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 9B U24 Sun Datacenter InfiniBand Switch 36 U20 10B U24 Sun Datacenter InfiniBand Switch 36 U20 11B U24 Sun Datacenter InfiniBand Switch 36 U20 8A U24 Sun Datacenter InfiniBand Switch 36 U20 9A U24 Sun Datacenter InfiniBand Switch 36 9A 10A 11A 8A 9B 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 243

244 SPARC SuperClusterT4-4 Oracle Exadata Storage 88 Oracle Exadata Storage InfiniBand ( ) InfiniBand U20 10A U24 Sun Datacenter InfiniBand Switch 36 U20 11A U24 Sun Datacenter InfiniBand Switch 36 10B 11B 2 QDR InfiniBand 2 QDR InfiniBand SPARC SuperClusterT4-4 Oracle Exadata Storage Sun Datacenter InfiniBand Switch 36. SPARC SuperCluster T4-4 Oracle Exadata Storage Oracle Exadata Storage Sun Datacenter InfiniBand Switch 36.,. InfiniBand QSFP(Quad Small Form-factor Pluggable)... Oracle Exadata Storage Sun Datacenter InfiniBand Switch 36 ( ) Oracle Exadata Storage. 245 SPARC SuperCluster T4-4 Oracle Exadata Storage. SPARC SuperClusterT4-4 InfiniBand SPARC SuperCluster T4-4 1(U1). 18(U18) 24(U24) SPARC SuperClusterT

245 SPARC SuperClusterT4-4 Oracle Exadata Storage Oracle Exadata Storage InfiniBand Oracle Exadata Storage 1(U1) (, Oracle Exadata Storage ). 20(U20) 24(U24). SPARC SuperClusterT4-4 Oracle Exadata Storage Oracle Exadata Storage Oracle SPARC SuperCluster T4-4. Oracle Exadata Storage SPARC SuperCluster T4-4. Oracle Exadata Storage Oracle Exadata Storage SPARC SuperCluster T SPARC SuperCluster T4-4 Oracle Exadata Storage Oracle Exadata Storage. 245

246 SPARC SuperClusterT4-4 Oracle Exadata Storage Oracle Exadata Storage SPARC SuperCluster T Oracle Exadata Storage 2. Oracle Exadata Storage. SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 Oracle Exadata Storage Oracle Exadata Storage. Oracle Exadata Storage SPARC SuperClusterT

247 SPARC SuperClusterT4-4 Oracle Exadata Storage. 1(R1) SPARC SuperCluster T4-4. 2(R2) Oracle Exadata Storage. InfiniBand InfiniBand. InfiniBand 1(IB1) SPARC SuperCluster T4-4 U1. InfiniBand 2(IB2). SPARC SuperCluster T4-4 U18 Oracle Exadata Storage U20 InfiniBand 3(IB3) U24. Oracle Exadata Storage SPARC SuperCluster T

248 SPARC SuperClusterT4-4 Oracle Exadata Storage 89 R1-IB3 2 R1-IB2 2 R1-IB3-P2A ~ R2-IB3-P2A R1-IB3-P2B ~ R2-IB3-P2B R1-IB3-P7B ~ R2-IB2-P7B R1-IB3-P12A ~ R2-IB2-P12A R1-IB2-P7B ~ R2-IB3-P7B R1-IB2-P12A ~ R2-IB3-P12A R1-IB2-P2A ~ R2-IB2-P2A R1-IB2-P2B ~ R2-IB2-P2B SPARC SuperClusterT4-4 Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage Oracle SPARC SuperCluster T4-4. Oracle Exadata Storage SPARC SuperCluster T SPARC SuperCluster T4-4 Oracle Exadata Storage ,. 248 SPARC SuperClusterT

249 SPARC SuperClusterT4-4 Oracle Exadata Storage, (R1) SPARC SuperCluster T4-4. 2(R2) Oracle Exadata Storage. InfiniBand InfiniBand. InfiniBand 1(IB1) U1. InfiniBand 2(IB2). SPARC SuperCluster T4-4 U18 Oracle Exadata Storage U20 InfiniBand 3(IB3) U24. (R1-IB1). 249