Copyright 2011, 2012, Oracle and/or its affiliates. All rights reserved.,.,,,,,,,,,,,,.,...,. U.S. GOVERNMENT END USERS. Oracle programs, including an

SPARC SuperCluster T4-4 : E27681 04 2012 12

Copyright 2011, 2012, Oracle and/or its affiliates. All rights reserved.,.,,,,,,,,,,,,.,...,. U.S. GOVERNMENT END USERS. Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government..., -,,. Oracle Corporation. Oracle Java Oracle Corporation /.. Intel Intel Xeon Intel Corporation. SPARC SPARC International, Inc.. AMD, Opteron, AMD AMD Opteron Advanced Micro Devices. UNIX The Open Group. ( ) 3,. Oracle Corporation 3,. Oracle Corporation 3,,. 130123@25097

...7...7...7...8...8...9 SPARC SuperCluster T4-4...9... 12... 16 SPARC SuperCluster T4-4... 29... 32 SPARC SuperCluster T4-4... 34... 41... 57... 57... 58... 60... 64... 67... 70... 73... 73 Oracle... 75... 75... 78... 81... 84 Fiber Channel... 95 3

SPARC SuperCluster T4-4... 97...97... 98... 100 Oracle Solaris 11... 101 Dynamic Intimate Shared Memory... 105... 106 Exadata Storage Server... 106 SPARC SuperCluster... 111 CPU... 113 SPARC SuperCluster T4-4... 123 SPARC SuperCluster T4-4... 123 OCM SPARC SuperCluster T4-4... 141 EM Exadata SPARC SuperCluster... 146 Exalogic... 147 Exalogic... 147 Exalogic... 147... 148... 149 Dept1_Cluster1 GridLink... 152 Dept1_Cluster1 SDP JDBC... 156 Exalogic SDP InfiniBand... 158... 163... 163 InfiniBand... 169... 175 Sun ZFS Storage 7320... 180 PDU... 180 3 PDU... 181 SPARC SuperCluster T4-4... 183... 183... 185... 187... 189... 192... 196 4 SPARC SuperClusterT4-4 2012 12

... 200... 206... 213 Oracle Exadata Storage... 213... 214 Oracle Exadata Storage... 225 IP... 225 ( )... 226 SPARC SuperCluster T4-4 Oracle Exadata Storage... 244... 277 5

,, Oracle SPARC SuperCluster T4-4.,. Oracle. Oracle. SPARC SuperCluster T4-4. 7 8 8. SPARC SuperCluster T4-4 SPARC T4-4. /opt/oracle/node/doc/e21659_01/index.html Oracle http://www.oracle.com/documentation 7

. http://www.oracle.com/goto/docfeedback My Oracle Support http://support.oracle.com : http://www.oracle.com/accessibility/support.html Oracle http://www.oracle.com/us/corporate/accessibility/index.html 8 SPARC SuperClusterT4-4 2012 12

SPARC SuperCluster T4-4.. 9 SPARC SuperCluster T4-4 12 16 29 SPARC SuperCluster T4-4 32 34 SPARC SuperCluster T4-4 41 SPARC SuperClusterT4-4 SPARC SuperCluster T4-4. SPARC SuperCluster T4-4, SPARC SuperCluster T4-4 Oracle Database 11g Real Application Clusters(Oracle RAC) Oracle Solaris Cluster.,. SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 (Online Transaction Processing, OLTP). Oracle Exadata Storage Server, Oracle SPARC SuperCluster T4-4. Oracle SPARC SuperCluster T4-4., SPARC SuperCluster T4-4. 9

SPARC SuperClusterT4-4 SPARC SuperCluster T4-4 Oracle. SPARC SuperCluster T4-4. Oracle Oracle Exadata Storage Server Oracle. Oracle Real Application Clusters Oracle Partitioning SPARC SuperCluster T4-4 SPARC SuperCluster T4-4,, InfiniBand. SPARC SuperCluster T4-4 10GbE( ) 1GbE( ). InfiniBand Exadata Exalogic SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 InfiniBand,,. Oracle SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 Oracle Exadata. SPARC T4-4 10GbE SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. 600GB 15K RPM SAS 1 3TB 7.2K RPM SAS 1 96GB Exadata 1 SPARC SuperClusterT4-4 SPARC SuperCluster T4-4.. SPARC SuperCluster T4-4.. SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 48 Cisco 4948.. 10 SPARC SuperClusterT4-4 2012 12

SPARC SuperClusterT4-4 1U 48. SPARC SuperCluster T4-4. Oracle,.,. Cisco 4948 CAT5.. Cisco 4948. Oracle Exadata Storage SPARC SuperCluster T4-4 Oracle Exadata, SPARC SuperCluster T4-4 Oracle Database(DB) Oracle Exadata. Exadata Storage Server SPARC SuperCluster T4-4 Oracle Exadata, SPARC SuperCluster T4-4 Oracle Exadata. Exadata Storage Server. Oracle Database Oracle Solaris 10. Oracle Oracle Solaris Oracle Solaris 10 Oracle Solaris 11. Oracle Exadata Storage Server, Exadata Storage Server. Exadata Storage Server. Exadata Storage Server. SPARC T4-4. Oracle.. Oracle.,.. InfiniBand. 11

SPARC SuperCluster T4-4 InfiniBand. Oracle Database,.,. Oracle Exadata Storage Server USB Exadata Storage Server. USB 100mA. SPARC T4-4 SCSI NFS Exadata Storage Server. FCoE(Fibre Channel Over Ethernet). SPARC SuperCluster T4-4, Oracle Exadata Oracle Exalogic Elastic Cloud InfiniBand. SPARC SuperCluster T4-4, Oracle Exadata Oracle Exalogic Elastic Cloud. SPARC SuperCluster T4-4 InfiniBand SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320.. 12 14 12 SPARC SuperClusterT4-4 2012 12

1 SPARC SuperCluster T4-4, 1: Exadata Storage Server (6) 2: SPARC T4-4 (4) 13

3: Sun ZFS Storage 7320 (2) 4: Sun Datacenter InfiniBand Switch 36 (2) 5: Sun Disk Shelf 6: Cisco Catalyst 4948 7: Sun Datacenter InfiniBand Switch 36 Oracle Exadata Storage. 213 Oracle Exadata Storage. SPARC SuperCluster T4-4 8 SPARC SuperCluster T4-4 Oracle Exadata Exalogic InfiniBand. 183 SPARC SuperCluster T4-4. 14 SPARC SuperClusterT4-4 2012 12

2 SPARC SuperCluster T4-4, 1: Sun ZFS Storage 7320 (2) 2: Sun Datacenter InfiniBand Switch 36 (2) 3: Sun Disk Shelf 15

4: Cisco Catalyst 4948 5: SPARC T4-4 (2) 6: Exadata Storage Server (3) 7: Sun Datacenter InfiniBand Switch 36 Oracle Exadata Storage. 213 Oracle Exadata Storage. SPARC SuperCluster T4-4 8 SPARC SuperCluster T4-4 Oracle Exadata Exalogic InfiniBand. 183 SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. 16 20 SPARC SuperCluster T4-4. 16 SPARC T4-4 18 Exadata Storage Server 19 Sun ZFS Storage 7320 19 Sun Datacenter InfiniBand Switch 36 20 Cisco Catalyst 4948 20 SPARCT4-4 SPARC SuperCluster T4-4 SPARC T4-4. SPARC SuperCluster T4-4 SPARC T4-4. SPARC T4-4. ( 20 SPARC T4-4 ) SPARC T4-4 16 SPARC SuperClusterT4-4 2012 12

SPARCT4-4 SPARC T4-4., SPARC T4-4. SPARC T4-4. SPARC T4-4. AC SPARC T4-4 AC.. SPARC T4-4 6 SSD. SPARC SuperCluster T4-4 6. SPARC T4-4.. 17

Exadata Storage Server Exadata Storage Server SPARC SuperCluster T4-4 6 Exadata Storage Server. SPARC SuperCluster T4-4 3 Exadata Storage Server. Exadata Storage Server. Exadata Storage Server. 23 Exadata Storage Server. Exadata Storage Server Exadata Storage Server. Exadata Storage Server. Exadata Storage Server 12 SPARC SuperCluster T4-4. SPARC SuperCluster 18 SPARC SuperClusterT4-4 2012 12

Sun Datacenter InfiniBand Switch 36 T4-4 Exadata Storage Server 12. 33. Sun ZFS Storage 7320 SPARC SuperCluster T4-4 Sun ZFS Storage 7320. Sun ZFS Storage 7320. Sun ZFS Storage 7320 2 Sun Disk Shelf 1 Sun ZFS Storage 7320. Sun ZFS Storage 7320 Sun Disk Shelf. 25 Sun ZFS Storage 7320. Sun ZFS Storage 7320 Sun ZFS Storage 7320 Sun Disk Shelf. Sun ZFS Storage 7320 Sun Disk Shelf. Sun ZFS Storage 7320. Sun Disk Shelf SPARC SuperCluster T4-4 20 (logzillas ) SSD. SPARC SuperCluster T4-4. 29 SPARC SuperCluster T4-4. Sun Datacenter InfiniBand Switch 36 SPARC SuperCluster T4-4 Sun Datacenter InfiniBand Switch 36... InfiniBand SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320. 20. 19

Cisco Catalyst 4948 Cisco Catalyst 4948 Cisco Catalyst 4948. Cisco Catalyst 4948. SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. 28. SPARC SuperCluster T4-4. 20 SPARC T4-4 23 Exadata Storage Server 25 Sun ZFS Storage 7320 28 SPARCT4-4 4 SPARC T4-4. 20 (SPARC T4-4 ) 21 InfiniBand (SPARC T4-4 ) 22 Oracle ILOM (SPARC T4-4 ) 23 1GbE (SPARC T4-4 ) 23 10GbE (SPARC T4-4 ) (SPARCT4-4 ) SPARC T4-4. 20 SPARC SuperClusterT4-4 2012 12

InfiniBand 개인 네트워크 물리적 연결(SPARC T4-4 서버) 그림 3 SPARC T4-4 서버의 카드 및 포트 위치 그림 범례 1: NET MGT 포트 - Oracle ILOM 관리 네트워크 연결용( 22 페이지 Oracle ILOM 관리 네트워크 물리적 연결(SPARC T4-4 서버) 참조) 2: NET0-NET3 포트 - 1GbE 호스트 관리 네트워크 연결용( 23 페이지 1GbE 호스트 관리 네트워크 물리적 연결(SPARC T4-4 서버) 참조) 3: 이중 포트 InfiniBand 호스트 채널 어댑터 - InfiniBand 네트워크 연결용( 21 페이지 InfiniBand 개인 네트워크 물리적 연결(SPARC T4-4 서버) 참조) 4: 이중 포트 10GbE 네트워크 인터페이스 카드 - 10GbE 클라이언트 액세스 네트워크 연결용( 23 페이지 10GbE 클라이언트 액세스 네트워크 물리적 연결(SPARC T4-4 서버) 참조) InfiniBand 개인 네트워크 물리적 연결(SPARC T4-4 서버) 각 SPARC T4-4 서버에는 이중 포트 Sun QDR InfiniBand PCIe 편평 호스트 채널 어댑터(HCA)가 네 개 포함되어 있습니다. InfiniBand HCA는 모든 SPARC T4-4 서버에서 다음 익스프레스 모듈 포트에 설치됩니다. 익스프레스 모듈 슬롯 1 익스프레스 모듈 슬롯 7 익스프레스 모듈 슬롯 9 익스프레스 모듈 슬롯 12 InfiniBand HCA 위치에 대한 자세한 내용은 20 페이지 카드 및 포트 위치(SPARC T4-4 서버) 를 참조하십시오. 21

Oracle ILOM (SPARCT4-4 ) InfiniBand HCA ( 1 2) SPARC T4-4. SPARC T4-4 InfiniBand. InfiniBand. SPARC T4-4 SPARC T4-4 InfiniBand. SPARC T4-4 InfiniBand IP. 29 SPARC SuperCluster T4-4. Oracle ILOM (SPARCT4-4 ) SPARC T4-4 SPARC T4-4 Oracle ILOM (NET MGT ) Oracle ILOM(Integrated Lights Out Manager). SPARC T4-4 Oracle ILOM IP. 22 SPARC SuperClusterT4-4 2012 12

InfiniBand (Exadata Storage Server) NET MGT 20 (SPARC T4-4 ). 1GbE (SPARCT4-4 ) SPARC T4-4 SPARC T4-4 1GbE (NET0 - NET3 ) 1GbE. 1GbE. 29 SPARC SuperCluster T4-4. 1GbE 20 (SPARC T4-4 ). 10GbE (SPARCT4-4 ) SPARC T4-4 Sun Dual 10GbE SFP+ PCIe 2.0 (NIC). 10GbE NIC SPARC T4-4. 2 6 10 14 10GbE NIC 20 (SPARC T4-4 ). 10GbE NIC ( 0 1). 10GbE NIC IPMP. 10GbE NIC IPMP. SPARC T4-4 10GbE. 29 SPARC SuperCluster T4-4. Exadata Storage Server Exadata Storage Server 3. 23 InfiniBand (Exadata Storage Server) 24 Oracle ILOM (Exadata Storage Server) 24 1GbE (Exadata Storage Server) InfiniBand (Exadata Storage Server) Exadata Storage Server Sun QDR InfiniBand PCIe (HCA). InfiniBand 23

Oracle ILOM (Exadata Storage Server) HCA. IP InfiniBand IP Exadata Storage Server. InfiniBand HCA Exadata Storage Server. Exadata Storage Server InfiniBand. Oracle ILOM (Exadata Storage Server) Exadata Storage Server Exadata Storage Server Oracle ILOM (NET MGT ) Oracle ILOM. Exadata Storage Server Oracle ILOM IP. 1GbE (Exadata Storage Server) Exadata Storage Server Exadata Storage Server 1GbE (NET 0 ) 1GbE. Exadata Storage Server 1GbE IP. 24 SPARC SuperClusterT4-4 2012 12

InfiniBand (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 Sun ZFS Storage 7320. 25 InfiniBand (Sun ZFS Storage 7320 ) 26 Oracle ILOM (Sun ZFS Storage 7320 ) 26 1GbE (Sun ZFS Storage 7320 ) 27 SAS (Sun ZFS Storage 7320 ) 28 (Sun ZFS Storage 7320 ) InfiniBand (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 Sun ZFS Storage 7320 InfiniBand. Sun ZFS Storage 7320 Sun Dual Port 40GB InfiniBand QDR HCA. InfiniBand HCA. IP InfiniBand IP Sun ZFS Storage 7320. InfiniBand HCA Sun ZFS Storage 7320. Sun ZFS Storage 7320 InfiniBand. 25

Oracle ILOM (Sun ZFS Storage 7320 ) Oracle ILOM (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 Sun ZFS Storage 7320 Oracle ILOM. NET0 Oracle ILOM. Oracle ILOM IP. 1GbE (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 Sun ZFS Storage 7320 1GbE. 1GbE. NET0( 25 ) NET1( 26 ) 1GbE IP. 26 SPARC SuperClusterT4-4 2012 12

SAS (Sun ZFS Storage 7320 ) SAS (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320 SAS-2 HBA. Sun Disk Shelf SIM Link In SIM Link Out. Sun Disk Shelf. 1 SAS-2 HBA Sun Disk Shelf SIM Link Out. 2 SAS-2 HBA Sun Disk Shelf SIM Link In. Sun Disk Shelf SAS. 4 Sun ZFS Storage 7320 SAS 1: 1 2: 2 3: Sun Disk Shelf 27

(Sun ZFS Storage 7320 ) (Sun ZFS Storage 7320 ) Sun ZFS Storage 7320... SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. SPARC T4-4 SPARC T4-4 AC. AC, AC. Exadata Storage Server Exadata Storage Server AC. AC AC. Sun ZFS Storage 7320 Sun ZFS Storage 7320 AC. AC AC. Sun Disk Shelf Sun Disk Shelf AC. AC AC. Sun Datacenter InfiniBand Switch 36 Sun Datacenter InfiniBand Switch 36 AC. AC AC. 28 SPARC SuperClusterT4-4 2012 12

SPARC SuperClusterT4-4 Cisco Catalyst 4948 Cisco Catalyst 4948 AC. AC AC. SPARC SuperClusterT4-4 SPARC SuperCluster T4-4 (LDom),. SPARC T4-4. SPARC SuperCluster T4-4. CPU - 25% - 100%. CPU. CPU SPARC T4-4. - 25% - 100%.. SPARC T4-4. PCI - PCI 1-4. PCI. PCI 16 SPARC T4-4. SPARC SuperCluster T4-4 LDom. SPARC SuperCluster T4-4 SPARC T4-4 1-4 LDom. LDom. Oracle Solaris 10 Oracle Solaris 11 SPARC SuperCluster T4-4. 30 30 30 10GbE 29

31 InfiniBand SPARC SuperCluster T4-4 SPARC T4-4 16.. 10GbE (NIC) - 10GbE. InfiniBand (HCA) - InfiniBand... SPARC T4-4. Oracle Integrated Lights Out Manager(ILOM) SPARC T4-4 Oracle ILOM. SPARC T4-4. 1 GbE SPARC T4-4 1GbE (NET0 - NET3).. SPARC T4-4 1GbE (NET0 - NET3) IP (IPMP). IPMP SPARC T4-4. 10GbE 10GbE SPARC T4-4. 10GbE NIC. 10GbE NIC SPARC T4-4. 30 SPARC SuperClusterT4-4 2012 12

InfiniBand InfiniBand InfiniBand InfiniBand SPARC T4-4, Sun ZFS Storage 7320 Exadata Storage Server. SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 SPARC T4-4 SPARC T4-4 InfiniBand. SPARC T4-4 InfiniBand SPARC T4-4. 31 InfiniBand 32 InfiniBand InfiniBand SPARC T4-4 InfiniBand. SPARC T4-4 Sun Datacenter InfiniBand Switch 36 SPARC T4-4 Exadata Storage Server, Sun Datacenter InfiniBand Switch 36 SPARC T4-4 Sun ZFS Storage 7320, Sun Datacenter InfiniBand Switch 36 SPARC T4-4 InfiniBand HCA. InfiniBand HCA SPARC T4-4. InfiniBand HCA InfiniBand. Oracle Solaris 10 Sun ZFS Storage 7320 InfiniBand 1 Oracle Database 11g Real Application Clusters(Oracle RAC) Exadata Storage Server InfiniBand 1 InfiniBand HCA SPARC T4-4 Sun Datacenter InfiniBand Switch 36. SPARC T4-4 21 InfiniBand (SPARC T4-4 ). 31

InfiniBand SPARC T4-4 (Oracle Solaris 10 Oracle Solaris 11 ) InfiniBand. SPARC T4-4 Sun Datacenter InfiniBand Switch 36 SPARC T4-4 Sun ZFS Storage 7320, Sun Datacenter InfiniBand Switch 36 Exadata Storage Server. SPARC T4-4 InfiniBand HCA. InfiniBand HCA SPARC T4-4. InfiniBand HCA InfiniBand. Sun ZFS Storage 7320 InfiniBand 1 Oracle Solaris Cluster InfiniBand 2 InfiniBand HCA SPARC T4-4 Sun Datacenter InfiniBand Switch 36. SPARC T4-4 21 InfiniBand (SPARC T4-4 ).. SPARC SuperCluster T4-4 SPARC T4-4 SPARC T4-4.. 32 SPARC SuperClusterT4-4 2012 12

Oracle Solaris SPARC SuperCluster T4-4. 33 33 Oracle Solaris Oracle Database 11g Real Application Clusters(Oracle RAC) Oracle. Oracle RAC SPARC T4-4 Oracle Clusterware. Oracle Clusterware Oracle. Oracle Clusterware Oracle RAC. Oracle Clusterware Oracle RAC. Oracle Oracle. Oracle RAC. Oracle RAC.. Oracle RAC Oracle Oracle. Oracle RAC. Oracle RAC Oracle.. Oracle RAC. Oracle Solaris Oracle Solaris Cluster Oracle Solaris. SPARC SuperCluster T4-4 Oracle Solaris Cluster SPARC T4-4 Oracle Solaris. 33

SPARC SuperClusterT4-4 SPARC SuperClusterT4-4 SPARC SuperCluster T4-4.. SPARC SuperCluster T4-4 SPARC T4-4 2 Exadata Storage Server 3 Sun ZFS Storage 7320 2 Sun Disk Shelf 1 Sun ZFS Storage 7320 1 Sun Datacenter InfiniBand Switch 36 3 48 Cisco Catalyst 4948 1 SPARC T4-4 1 Oracle Solaris 10 1 SPARC T4-4 50% CPU SPARC T4-4 50% SPARC T4-4 PCI. 34 37 Oracle Solaris 10 SPARC T4-4. 34 CPU, PCI 35 InfiniBand 36 10GbE CPU, PCI SPARC T4-4. CPU - 0 0 1-0 0 1 lgroup 34 SPARC SuperClusterT4-4 2012 12

InfiniBand PCI - 0 PCI pci@400 pci@500 InfiniBand InfiniBand SPARC T4-4. SPARCT4-4 SPARC T4-4 4 Sun QDR InfiniBand PCIe HCA 2 SPARC T4-4. 1 9 InfiniBand HCA IPoIB (InfiniBand HCA IPoIB ). InfiniBand 35

10GbE HCA SPARC T4-4. 23 InfiniBand (Exadata Storage Server). SPARC SuperClusterT4-4 SPARC T4-4 InfiniBand, Exadata Storage Server Sun ZFS Storage 7320 InfiniBand. 10GbE SPARC T4-4 4 Sun Dual 10GbE SFP+ PCIe 2.0 NIC 2 SPARC T4-4. 2 10 36 SPARC SuperClusterT4-4 2012 12

CPU, PCI 10GbE NIC IPMP. IPMP SPARC SuperCluster T4-4. 1 IPMP 10GbE. 2 10GbE NIC - 0 10 10GbE NIC - 1 SPARC SuperCluster T4-4 SPARC T4-4 10GbE. Oracle Solaris 10 SPARC T4-4 Oracle Solaris 10. 37 CPU, PCI 38 InfiniBand 40 10GbE CPU, PCI SPARC T4-4 Oracle Solaris 10. CPU - 1 2 3-1 2 3 lgroup PCI - 1 PCI pci@600 pci@700 CPU, PCI Oracle Solaris 10. 37

InfiniBand InfiniBand InfiniBand SPARC T4-4 Oracle Solaris 10. SPARCT4-4 SPARC T4-4 4 Sun QDR InfiniBand PCIe HCA 2 Oracle Solaris 10 SPARC T4-4. 7 12 Oracle Solaris 10 InfiniBand. 38 SPARC SuperClusterT4-4 2012 12

InfiniBand InfiniBand HCA Oracle Solaris 10 IPoIB. InfiniBand HCA Oracle Solaris Cluster. Oracle Solaris 10 InfiniBand. 7 InfiniBand HCA - 1 12 InfiniBand HCA - 2 SPARC SuperClusterT4-4 SPARC T4-4 Oracle Solaris 10 InfiniBand Sun ZFS Storage 7320 InfiniBand. Oracle Solaris 10, Exadata Storage Server. 39

10GbE 10GbE SPARC T4-4 Sun Dual 10GbE SFP+ PCIe 2.0 NIC Oracle Solaris 10 SPARC T4-4. 6 14 10GbE NIC IPMP. IPMP SPARC SuperCluster T4-4. 1 IPMP Oracle Solaris 10 10GbE. 6 10GbE NIC - 0 14 10GbE NIC - 1 40 SPARC SuperClusterT4-4 2012 12

SPARC SuperCluster T4-4. 41 44 SPARC SuperCluster T4-4 45 IP SPARC SuperCluster T4-4 SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320 SPARC T4-4. SPARC T4-4. SPARC T4-4. 4 (NET0, NET1, NET2 NET3) Oracle ILOM(Oracle Integrated Lights Out Manager) (NET MGT) 1 InfiniBand Sun QDR InfiniBand PCIe (Host Channel Adapter, HCA) 4 10GbE Sun Dual 10GbE SFP+ PCIe 2.0 (NIC) 4 10GbE PCIe 2.0 QSFP. Exadata Storage Server. 1 (NET0) 41

InfiniBand Sun QDR InfiniBand PCIe (Host Channel Adapter, HCA) 1 Oracle ILOM (NET MGT) 1 Sun ZFS Storage 7320. 1 NET0( 25 ) NET1( 26 ) InfiniBand QDR InfiniBand 1 Oracle ILOM 1 (NET0) Oracle ILOM. SPARC SuperCluster T4-4 Cisco Catalyst 4948. IP. IP NTP VLAN(Virtual Local Area Network),. SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. SPARC SuperCluster T4-4... SPARC SuperCluster T4-4.,, Oracle ILOM.. 42 SPARC SuperClusterT4-4 2012 12

PDU. SPARC T4-4 Exadata Storage Server. 1GbE Oracle ILOM Oracle Integrated Lights Out Manager. SPARC T4-4 4 1GbE (NET 0 - NET3). NET IPMP. 29 SPARC SuperCluster T4-4. Sun ZFS Storage 7320. 1: 1GbE Oracle ILOM NET0. 2: Oracle ILOM NET0 1GbE NET1. SPARC SuperCluster T4-4 Oracle ILOM 1GbE. SPARC T4-4 1GbE.. 10GbE SPARC T4-4. SCAN(Single Client Access Name) Oracle RAC IP(Virtual IP, VIP). InfiniBand InfiniBand SPARC T4-4, Sun ZFS Storage 7320 Exadata Storage Server. SPARC T4-4 Oracle Oracle RAC Exadata Storage Server Sun ZFS Storage 7320. SPARC T4-4 Oracle Solaris Cluster Sun ZFS Storage 7320. 43

SPARC SuperClusterT4-4 SPARC SuperCluster T4-4.... 5 SPARC SuperCluster T4-4 SPARC SuperClusterT4-4 SPARC SuperCluster T4-4. 1 1 44 SPARC SuperClusterT4-4 2012 12

IP ( ) 1 ( ) SPARC T4-4 1-2 : : 4-8 : 2-4. (SPARC T4-4 1 IPMP.) IP (SPARC SuperClusterT4-4) SPARC SuperCluster T4-4 IP. 45 IP 45 IP ( ) 50 IP ( ) IP 4 IP. IP SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320 Oracle ILOM IP. IP SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320 IP. InfiniBand IP InfiniBand SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320. SPARC SuperCluster T4-4 InfiniBand SPARC SuperCluster T4-4 Oracle Exadata Exalogic InfiniBand 1 SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 Oracle Exadata Exalogic SPARC T4-4. 10GbE IP 10GbE IP. 5. IP ( ) SPARC SuperCluster T4-4 IP. 46 Oracle ILOM IP ( ) 48 InfiniBand 10GbE IP ( ) 45

IP ( ) Oracle ILOM IP ( ) 2 Oracle ILOM IP ( ) ( ) Oracle ILOM Oracle ILOM IP IP PDU-A( ) sscpdua 192.168.1.210 PCU-B( ) sscpdub 192.168.1.211 42 Exadata Storage Server 6 ssces6-sp 192.168.1.106 cell06 192.168.1.6 41 40 Exadata Storage Server 5 ssces5-sp 192.168.1.105 cell05 192.168.1.5 39 38 Exadata Storage Server 4 ssces4-sp 192.168.1.104 cell04 192.168.1.4 37 36 SPARC T4-4 4 ssccn4-sp 192.168.1.111 ssccn4-m4 192.168.1.41 35 ssccn4-m3 192.168.1.31 34 ssccn4-m2 192.168.1.21 33 ssccn4-m1 192.168.1.11 32 31 SPARC T4-4 3 ssccn3-sp 192.168.1.110 ssccn3-m4 192.168.1.40 30 ssccn3-m3 192.168.1.30 29 ssccn3-m2 192.168.1.20 28 ssccn3-m1 192.168.1.10 27 26 Sun ZFS Storage 7320 2 25 Sun ZFS Storage 7320 1 24 Sun Datacenter InfiniBand Switch 36( 2) sscsn2-sp 192.168.1.116 sscsn2-m1 192.168.1.16 sscsn1-sp 192.168.1.115 sscsn1-m1 192.168.1.15 sscnm1-m3 192.168.1.203 46 SPARC SuperClusterT4-4 2012 12

IP ( ) 2 Oracle ILOM IP ( ) ( ) ( ) Oracle ILOM Oracle ILOM IP IP 23 Sun ZFS Storage 7320 22 Sun Disk Shelf 21 20 19 Cisco Catalyst 4948 18 Sun Datacenter InfiniBand Switch 36( 1) ssc4948-m1 192.168.1.200 sscnm1-m2 192.168.1.202 17 SPARC T4-4 2 ssccn2-sp 192.168.1.109 ssccn2-m4 192.168.1.49 16 ssccn2-m3 192.168.1.39 15 ssccn2-m2 192.168.1.29 14 ssccn2-m1 192.168.1.9 13 12 SPARC T4-4 1 ssccn1-sp 192.168.1.108 ssccn1-m4 192.168.1.38 11 ssccn1-m3 192.168.1.28 10 ssccn1-m2 192.168.1.18 9 ssccn1-m1 192.168.1.8 8 7 Exadata Storage Server 3 ssces3-sp 192.168.1.103 cell03 192.168.1.3 6 5 Exadata Storage Server 2 ssces2-sp 192.168.1.102 cell02 192.168.1.2 4 3 Exadata Storage Server 1 ssces1-sp 192.168.1.101 cell01 192.168.1.1 2 1 Sun Datacenter InfiniBand Switch 36( ) sscnm1-m1 192.168.1.201 47

IP ( ) InfiniBand 10GbE IP ( ) 3 InfiniBand 10GbE IP ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP PDU-A( ) PCU-B( ) 42 Exadata Storage Server 6 ssces6-stor 192.168.10.6 41 40 Exadata Storage Server 5 ssces5-stor 192.168.10.5 39 38 Exadata Storage Server 4 ssces4-stor 192.168.10.4 37 36 SPARC T4-4 4 ssccn4-ib4 192.168.10.41 ssccn4-tg8 ssccn4-tg7 35 ssccn4-ib3 192.168.10.31 ssccn4-tg6 ssccn4-tg5 34 ssccn4-ib2 192.168.10.21 ssccn4-tg4 ssccn4-tg3 33 ssccn4-ib1 192.168.10.11 ssccn4-tg2 32 ssccn4-tg1 31 SPARC T4-4 3 ssccn3-ib4 192.168.10.40 ssccn3-tg8 ssccn3-tg7 30 ssccn3-ib3 192.168.10.30 ssccn3-tg6 ssccn3-tg5 29 ssccn3-ib2 192.168.10.20 ssccn3-tg4 ssccn3-tg3 28 ssccn3-ib1 192.168.10.10 ssccn3-tg2 27 ssccn3-tg1 192.168.40.32 192.168.40.31 192.168.40.30 192.168.40.29 192.168.40.28 192.168.40.27 192.168.40.26 192.168.40.25 192.168.40.24 192.168.40.23 192.168.40.22 192.168.40.21 192.168.40.20 192.168.40.19 192.168.40.18 192.168.40.17 48 SPARC SuperClusterT4-4 2012 12

IP ( ) 3 InfiniBand 10GbE IP ( ) ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP 26 Sun ZFS Storage 7320 2 25 Sun ZFS Storage 7320 1 24 Sun Datacenter InfiniBand Switch 36( 2) 23 Sun ZFS Storage 7320 22 Sun Disk Shelf sscsn1-stor2 sscsn1-stor1 192.168.30.15 21 20 19 Cisco Catalyst 4948 18 Sun Datacenter InfiniBand Switch 36( 1) 17 SPARC T4-4 2 ssccn2-ib4 192.168.10.39 ssccn2-tg8 ssccn2-tg7 16 ssccn2-ib3 192.168.10.29 ssccn2-tg6 ssccn2-tg5 15 ssccn2-ib2 192.168.10.19 ssccn2-tg4 ssccn2-tg3 14 ssccn2-ib1 192.168.10.9 ssccn2-tg2 13 ssccn2-tg1 192.168.40.16 192.168.40.15 192.168.40.14 192.168.40.13 192.168.40.12 192.168.40.11 192.168.40.10 192.168.40.9 49

IP ( ) 3 InfiniBand 10GbE IP ( ) ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP 12 SPARC T4-4 1 ssccn1-ib4 192.168.10.38 ssccn1-tg8 ssccn1-tg7 11 ssccn1-ib3 192.168.10.28 ssccn1-tg6 ssccn1-tg5 10 ssccn1-ib2 192.168.10.18 ssccn1-tg4 ssccn1-tg3 9 ssccn1-ib1 192.168.10.8 ssccn1-tg2 192.168.40.8 192.168.40.7 192.168.40.6 192.168.40.5 192.168.40.4 192.168.40.3 192.168.40.2 8 ssccn1-tg1 192.168.40.1 7 Exadata Storage Server 3 ssces3-stor 192.168.10.3 6 5 Exadata Storage Server 2 ssces2-stor 192.168.10.2 4 3 Exadata Storage Server 1 ssces1-stor 192.168.10.1 2 1 Sun Datacenter InfiniBand Switch 36( ) IP ( ) SPARC SuperCluster T4-4 IP. 51 Oracle ILOM IP ( ) 53 InfiniBand 10GbE IP ( ) 50 SPARC SuperClusterT4-4 2012 12

IP ( ) Oracle ILOM IP ( ) 4 Oracle ILOM IP ( ) ( ) Oracle ILOM Oracle ILOM IP IP PDU-A( ) sscpdua 192.168.1.210 PCU-B( ) sscpdub 192.168.1.211 42 4U 41 40 39 38 4U 37 36 35 34 4U 33 32 31 30 4U 29 28 27 26 Sun ZFS Storage 7320 2 25 Sun ZFS Storage 7320 1 24 Sun Datacenter InfiniBand Switch 36( 2) sscsn2-sp 192.168.1.116 sscsn2-m1 192.168.1.16 sscsn1-sp 192.168.1.115 sscsn1-m1 192.168.1.15 sscnm1-m3 192.168.1.203 51

IP ( ) 4 Oracle ILOM IP ( ) ( ) ( ) Oracle ILOM Oracle ILOM IP IP 23 Sun ZFS Storage 7320 22 Sun Disk Shelf 21 20 19 Cisco Catalyst 4948 18 Sun Datacenter InfiniBand Switch 36( 1) ssc4948-m1 192.168.1.200 sscnm1-m2 192.168.1.202 17 SPARC T4-4 2 ssccn2-sp 192.168.1.109 ssccn2-m4 192.168.1.49 16 ssccn2-m3 192.168.1.39 15 ssccn2-m2 192.168.1.29 14 ssccn2-m1 192.168.1.9 13 12 SPARC T4-4 1 ssccn1-sp 192.168.1.108 ssccn1-m4 192.168.1.38 11 ssccn1-m3 192.168.1.28 10 ssccn1-m2 192.168.1.18 9 ssccn1-m1 192.168.1.8 8 7 Exadata Storage Server 3 ssces3-sp 192.168.1.103 cell03 192.168.1.3 6 5 Exadata Storage Server 2 ssces2-sp 192.168.1.102 cell02 192.168.1.2 4 3 Exadata Storage Server 1 ssces1-sp 192.168.1.101 cell01 192.168.1.1 2 1 Sun Datacenter InfiniBand Switch 36( ) sscnm1-m1 192.168.1.201 52 SPARC SuperClusterT4-4 2012 12

IP ( ) InfiniBand 10GbE IP ( ) 5 InfiniBand 10GbE IP ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP PDU-A( ) PCU-B( ) 42 4U 41 40 39 38 4U 37 36 35 34 4U 33 32 31 30 4U 29 28 27 26 Sun ZFS Storage 7320 2 25 Sun ZFS Storage 7320 1 24 Sun Datacenter InfiniBand Switch 36( 2) sscsn1-stor2 sscsn1-stor1 192.168.30.15 53

IP ( ) 5 InfiniBand 10GbE IP ( ) ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP 23 Sun ZFS Storage 7320 22 Sun Disk Shelf 21 20 19 Cisco Catalyst 4948 18 Sun Datacenter InfiniBand Switch 36( 1) 17 SPARC T4-4 2 ssccn2-ib4 192.168.10.39 ssccn2-tg8 ssccn2-tg7 16 ssccn2-ib3 192.168.10.29 ssccn2-tg6 ssccn2-tg5 15 ssccn2-ib2 192.168.10.19 ssccn2-tg4 ssccn2-tg3 14 ssccn2-ib1 192.168.10.9 ssccn2-tg2 13 ssccn2-tg1 12 SPARC T4-4 1 ssccn1-ib4 192.168.10.38 ssccn1-tg8 ssccn1-tg7 11 ssccn1-ib3 192.168.10.28 ssccn1-tg6 ssccn1-tg5 10 ssccn1-ib2 192.168.10.18 ssccn1-tg4 ssccn1-tg3 9 ssccn1-ib1 192.168.10.8 ssccn1-tg2 8 ssccn1-tg1 192.168.40.16 192.168.40.15 192.168.40.14 192.168.40.13 192.168.40.12 192.168.40.11 192.168.40.10 192.168.40.9 192.168.40.8 192.168.40.7 192.168.40.6 192.168.40.5 192.168.40.4 192.168.40.3 192.168.40.2 192.168.40.1 7 Exadata Storage Server 3 ssces3-stor 192.168.10.3 6 54 SPARC SuperClusterT4-4 2012 12

IP ( ) 5 InfiniBand 10GbE IP ( ) ( ) ( ) InfiniBand InfiniBand IP 10GbE IP 10GbE IP 5 Exadata Storage Server 2 ssces2-stor 192.168.10.2 4 3 Exadata Storage Server 1 ssces1-stor 192.168.10.1 2 1 Sun Datacenter InfiniBand Switch 36( ) 55

. 57 58 60 64 67 70..... 10ohm..,. ( )48. ( ). 57

58 58 59 SPARC SuperCluster T4-4. 1998mm 78.66 600mm 23.62 ( ) 1200mm 47.24 ( ) 1112mm 43.78 2300mm 90 914mm 36 ( ) 862kg 1,900 ( ) 594kg 1,310 58 59 67. 914mm 36 914mm 36 914mm 36 58 SPARC SuperClusterT4-4 2012 12

58 59...,2. 280mm(11 ). 79 ( ).. 6 1: 113mm(4.4). 59

2: 374mm(14.72 ). 3: 113mm(4.4). 4: 1120mm(44.1 ). 5: 330mm(13 ). 6: 160mm(6.3 ). 7: 280mm(11 ). 67 58 58 60 60 61 61 PDU 60 61 61 PDU 15.81kVA 15.02kW 13.80kVA 13.11kW 8.97kVA 8.52kW 6.9kVA 6.5kW. PDU AC.. 60 SPARC SuperClusterT4-4 2012 12

PDU AC., (Uninterruptible Power Supply, UPS),. 60 61 61 PDU.. IEC.. 60 60 61 PDU PDU SPARC SuperCluster T4-4. 3 Oracle. SPARC SuperCluster T4-4 2 (Power Distribution Unit, PDU). 2 PDU. 6 PDU 1 7 61

PDU 6 PDU ( ) 3 8 1 9 3 10 7 SPARC SuperCluster T4-4 PDU kva 22kVA 7100873 7018123 PDU Amp 1ph 200-240VAC 110.4A(3 x36.8a) 42 C13 6 C19 3 x 50A, 1ph 36.8A ( ) Hubbell CS8264C PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 8 SPARC SuperCluster T4-4 3 PDU 3 kva 24kVA 6444A 597-1162-01 3ph 200 ~ 240VAC PDU Amp 120A(6 x 20A) 62 SPARC SuperClusterT4-4 2012 12

PDU 8 SPARC SuperCluster T4-4 3 PDU ( ) 3 42 C13 6 C19 2 x 60A, 3ph 34.6A ( ) IEC 309-3P4W-IP67 (60A, 250V, AC, 3ph) PDU Hubbell HBL460R/C9W IEC309 60A 3ph 4 PDU 4 kva 2m(6.6 ) PDU 4m(13 ). 9 SPARC SuperCluster T4-4 PDU kva 22kVA 7100874 7018124 1ph 200 ~ 240VAC PDU Amp 96A(3 x 32A) 42 C13 6 C19 3 x 32A, 1ph 32A ( ) IEC 309-2P3W-IP44 (32A, 250V, AC, 3ph) PDU Hubbell HBL332R/C9W IEC309 32A 1ph 3 PDU 6 kva 2m(6.6 ) PDU 4m(13 ). 63

10 SPARC SuperCluster T4-4 3 PDU 3 kva 24kVA 6445A 597-1163-01 PDU Amp 3ph 220/380 ~ 240/415VAC 109A(6 x18.1a) 42 C13 6 C19 2 x25a, 3ph 18A ( ) IEC 309-4P5W-IP44 (32A, 400V, AC, 3ph) PDU Hubbell HBL532R/C9W IEC309 32A 3ph 5 PDU 4 kva 2m(6.6 ) PDU 4m(13 ). 60 61 60 64 65 67,. 64 SPARC SuperClusterT4-4 2012 12

5 ~ 32 C (41 ~ 89.6 F) -40 ~ 70 C (-40 ~ 158 F) 21 ~ 23 C (70 ~ 47 F). 10 ~ 90%, 93% 45 ~ 50%,. 3048m(10000 ) 12000m(40000 ) 900m 300m 1. 65 67 53,966 BTU/ (56,937 kj/ ) 30,610 BTU/ (32,295 kj/ )..... 914mm(36 ), 914mm(36 )... 65

7 11 ( ) CFM 2,498 2,180 CFM 1,417 1,090 64 67 66 SPARC SuperClusterT4-4 2012 12

,. 400CFM... 6 4 65 64 59. 67 68 68 69 SPARC SuperCluster T4-4. 2159mm 8 1219mm 48 1575mm 62 ( ) 1009.2kg 222 67

( ) 721kg 1586 68 68 69 58..... :. 24. 67 68 69 58...,. 68 SPARC SuperClusterT4-4 2012 12

2184mm (86 ) 2040mm (80.32 ) 1220 (48 ) 600mm (23.62 ) 1575mm (62 ) 1200mm (47.24 ) 6 6, ( ) 1134kg (2500 ) 1134kg (2500 ) 67 68 69 58... 67 68 68 58 69

SPARC SuperCluster T4-4. 70 70 IP 71 DNS. ( ) 1Gb 10Gb 1 x 1GB 1 SPARC T4-4 1 IPMP. SPARC T4-4 1 4 x 10GB 1 SPARC T4-4 2 8 x 10GB SPARC T4-4 4 32 x 10GB 1 x 1GB SPARC T4-4 1 2 x 10GB 1 SPARC T4-4 2 4 x 10GB SPARC T4-4 4 16 x 10GB 70 IP 71 DNS 41 IP SPARC SuperCluster T4-4 IP. IP SPARC SuperCluster T4-4. 70 71 DNS 41 70 SPARC SuperClusterT4-4 2012 12

IP DNS.. GNS(Grid Naming Service). 1 2. SPARC SuperCluster T4-4 IP DNS(Domain Name System)., SCAN(Single Client Access Name) VIP DNS. IP SCAN. 3 DNS. ( DNS) DNS DNS. SCAN SCAN DNS. Oracle Grid Infrastructure Installation Guide for Linux - SCAN DNS - 70 IP 70 71

SPARC SuperCluster T4-4. 73 75 Oracle 75 78 81 84 95 Fiber Channel. 1.. 1.,,. 2.. 3.. 75 Oracle 57 SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 2.. 1. SPARC SuperCluster T4-4. 2. SPARC SuperCluster T4-4. 3.. 4.. 75. 3. PDU. 73

1.. 84 2. PDU(PDU A) 6 PDU. 3. Oracle ILOM 3-5. 4. SPARC SuperCluster T4-4 SPARC T4-4. 5.. 4. Sun ZFS Storage 7320. 1. 2 Sun ZFS Storage 7320. 2. Sun Storage 7320 NFS, 3-5. 3. Sun ZFS Storage 7320 IP ping. 5. Exadata Storage. Oracle ILOM Oracle ILOM Exadata Storage Server. 6. SPARC T4-4. 1. SPARC T4-4. 2.. 3. Sun Datacenter Infiniband Switch 36. 7. SPARC SuperCluster T4-4. 8.. ( ). 95 Fiber Channel 74 SPARC SuperClusterT4-4 2012 12

Oracle Oracle Oracle.. Important Safety Information for Sun Hardware Systems(821-1590). Sun Rack II Safety and Compliance Guide(820-4762). http://www.oracle.com/documentation. Sun Rack II Power Distribution Units Users Guide(820-4760). http://www.oracle.com/documentation.. 75 75 76. 2.. 8. 75

8. 76 SPARC SuperClusterT4-4 2012 12

... 1 2 3 4 2.... (Customer Information Sheet, CIS). 2. 12mm. 17mm. 2... 5. a... b.. 6 7. 67.. 65. 77

78 SPARC SuperCluster T4-4 79 ( ) 1 2 3 SPARC SuperClusterT4-4. 68. SPARC SuperClusterT4-4. 4... 0.6( 2 ). 4 SPARC SuperClusterT4-4...... 78 SPARC SuperClusterT4-4 2012 12

랙을 제자리로 이동 접지 케이블 설치(선택 사항) SPARC SuperCluster T4-4 배전 장치(Power Distribution Unit, PDU)는 해당 전원 코드를 통해 접지됩니다. 추가로 접지하려면 섀시 접지 케이블을 SPARC SuperCluster T4-4에 연결합니다. 추가 접지 지점을 사용하면 전류 누설을 좀 더 효과적으로 방출할 수 있습니다. 79

PDU.PDU.. 1 2 3.. 61.... 80 SPARC SuperClusterT4-4 2012 12

4.. 82 82 ( ) 81

4. 1 2 12mm..,4. ( ). 9. 1 4. 10mm.. Oracle. 82 SPARC SuperClusterT4-4 2012 12

9 1: 113mm(4.4). 2: 374mm(14.72 ). 3: 113mm(4.4). 4: 1120mm(44.1 ). 5: 330mm(13 ). 6: 160mm(6.3 ). 7: 280mm(11 ). 2 3 4... 83

5. 6 SPARC SuperClusterT4-4. 84 87 SPARC SuperCluster T4-4 91 SPARC SuperCluster T4-4 93 10GbE 1. 84 SPARC SuperClusterT4-4 2012 12

AC. 2 3. PDU. PDU. PDU-A. PDU-B. PDU 1 6 ( ). 10 PDU 85

1:. 2:. 4 5 6.... 86 SPARC SuperClusterT4-4 2012 12

7 8. PDU. 1 2 3 SPARC SuperClusterT4-4.. B(PDU B). PDU A. PDU B.. 87

PDU B ON( ). PDU B. 4. 12. PDU B LED. a. SPARCT4-4. LED LED PS1 PS3 LED PS0 PS2 LED LED SPARC T4-4. b. Sun ZFS Storage 7320. LED( ). LED ( 10 ). 88 SPARC SuperClusterT4-4 2012 12

LED( )- ( )- ( )- LED Sun ZFS Storage 7320. c. Sun Disk Shelf,. LED. LED ( )- ( )- d. Sun Datacenter InfiniBand Switch 36. LED(PS0 LED) - LED(PS1 LED) - e. Cisco Catalyst 4948. PS1 LED PS2 LED 4 A(PDU A). PDU A. 89

PDU A ON( ). PDU A LED. a. SPARCT4-4. LED LED PS1 PS3 LED PS0 PS2 LED b. Sun ZFS Storage 7320. LED LED - - Sun ZFS Storage 7320 IP ping. NET0 IP 45 IP..IP ping 91 SPARC SuperCluster T4-4. 90 SPARC SuperClusterT4-4 2012 12

c. Sun Disk Shelf. LED - - d. Exadata Storage Server. LED - Oracle ILOM ( 3 ). LED e. Sun Datacenter InfiniBand Switch 36. LED(PS0 LED) - LED(PS1 LED) - f. Cisco Catalyst 4948. PS1 LED PS2 LED 1 2 3 4 SPARC SuperClusterT4-4 USB. 2 Category 5E 1.. Cisco Catalyst 4948. SPARC SuperCluster T4-4 19... USB- USB Cisco. USB- (Sun Network QDR InfiniBand ). SPARC SuperCluster T4-4. 91

5. Windows. a. >... b. (TCP/IP).. (TCP/IP). c. IP IP. IP Cisco. SPARC T4-4 NET0 IP Oracle SPARC SuperCluster T4-4 IP. NET0 IP 45 IP. d. IP. e.. Linux. a. root. b. ETH0 SPARC SuperClusterT4-4. # ifconfig -a SPARC SuperCluster T4-4. c. ifconfig. # ifconfig eth0 192.0.2.10 netmask 255.255.255.0 up ifconfig 255.255.255.0 IPv4 192.0.2.10 eth0. 6 PuTTY telnet ssh. SPARC T4-4 IP IP. 92 SPARC SuperClusterT4-4 2012 12

Cisco Catalyst 4948 SPARC SuperCluster T4-4 NET0 IP. IP 45 IP. Oracle Oracle SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 IP IP. Oracle Oracle SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 IP. 10GbE IP DNS(Domain Name System)., SCAN(Single Client Access Name) VIP DNS. 10GbE SPARC SuperCluster T4-4. SPARC SuperCluster T4-4. PCIE 2, 6, 10 14, SPARC T4-4 Sun Dual 10GbE SFP+ PCIe NIC 4 ( 20 (SPARC T4-4 ) ) 10GbE NIC 10m SFP-QSFP 4 8 SFP-QSFP 10GbE. QSFP 10GbE ( : Sun Network 10GbE Switch 72p) 10GbE ( 4 8 ) SFP-QSFP 10GbE. 10GbE SPARC T4-4 SFP+ 10GbE 93

10GbE 10GbE 10Gb 1Gb SPARC SuperCluster T4-4 10GbE. SPARC SuperCluster T4-4 10GbE. 1 2 3 4 Sun Dual 10GbE SFP+ PCIe NIC. NIC SPARC SuperCluster T4-4. 23 10GbE (SPARC T4-4 ) 29 SPARC SuperCluster T4-4. 10GbE SFP. SFP 10 SFP-QSFP. 10GbE NIC SFP. 10GbE. 10GbE. 2 SFP-QSFP QSFP 2 10GbE QSFP ( Sun Network 10GbE Switch 72p). 2 SFP-QSFP SFP+ 16 SPARC T4-4 10GbE NIC SFP+. 10GbE NIC SFP+. 94 SPARC SuperClusterT4-4 2012 12

선택적 Fiber Channel 익스프레스 모듈 사용 그림 11 10GbE 클라이언트 액세스 네트워크에 대한 연결 예 그림 범례 1: QSFP 연결이 있는 10GbE 스위치(Sun Network 10GbE Switch 72p 표시) 2: SFP-QSFP 케이블의 QSFP 커넥터 끝, 10GbE 스위치의 QSFP 포트에 연결 3: SFP-QSFP 케이블의 SFP+ 커넥터 끝, SPARC T4-4 서버의 10GbE NIC의 SFP+ 포트에 연결 선택적 Fiber Channel 익스프레스 모듈 사용 선택적 Fiber Channel 익스프레스 모듈은 기존 저장 부속 시스템에서 SPARC SuperCluster T4-4와 통합된 Exadata Storage Server로의 데이터 마이그레이션을 용이하게 하는 데 사용할 수 있습니다. 선택적 Fiber Channel 익스프레스 모듈은 표준 SPARC SuperCluster T4-4 구성에 포함되어 있지 않으므로 개별적으로 구입해야 합니다. 지원되는 옵션은 다음과 같습니다. 95

Fiber Channel ExpressModule SG-XPCIEFCGBE-E8-N Sun StorageTek Dual 8Gb FC Dual 1GbE HBA, RoHS-6, Emulex X- ExpressModule SG-XPCIEFCGBE-Q8-N Sun StorageTek Dual 8Gb FC Dual 1GbE HBA, RoHS-6, Qlogic. X- Fiber Channel 10GbE NIC InfiniBand HCA. 0 3 4 5 8 11 13 15 SPARC T4-4 20 (SPARC T4-4 ). SPARC T4-4. SPARC SuperCluster T4-4. 29 SPARC SuperCluster T4-4. Fiber Channel. Oracle Solaris 10 Oracle Solaris 11 Fiber Channel Oracle Database 11gR2. Fiber Channel Oracle Database 11gR2. Oracle Fiber Channel GbE. Oracle.. SPARC T4-4 : http://docs.oracle.com/cd/e23411_01/html/e23412/ 96 SPARC SuperClusterT4-4 2012 12

SPARC SuperClusterT4-4 SPARC SuperCluster T4-4. 97 98 100 101 Oracle Solaris 11 105 Dynamic Intimate Shared Memory 106 106 Exadata Storage Server 111 SPARC SuperCluster 113 CPU SPARC SuperCluster T4-4....... 97

98 100. 1. Oracle Solaris Cluster. # /usr/cluster/bin/cluster shutdown -g 0 -y 2. Ops Center. # /opt/sunwxvmoc/bin/ecadm ha-stop-no-relocate 3.. http://docs.oracle.com/cd/b28359_01/server.111/b28310/start003.htm 4. 98 Exadata Storage Server. 5. 99 Exadata Storage Server 6. 99 LDom 7. SPARC T4-4. # init 0 8. Sun ZFS Storage 7320.. 9.. 100. Exadata Storage Server Exadata Storage Server. Exadata. http://wd0338.oracle.com/ archive/cd_ns/e13877_01/doc/doc.112/e13874/maintenance.htm#cegbhcjg 1. CellCLI > LIST GRIDDISK ATTRIBUTES name WHERE asmdeactivationoutcome!= Yes Oracle ASM Exadata Storage Server. 98 SPARC SuperClusterT4-4 2012 12

Exadata Storage Server Oracle ASM. 2 3 4 Exadata Storage Server. CellCLI> ALTER GRIDDISK ALL INACTIVE. INACTIVE. CellCLI> LIST GRIDDISK WHERE STATUS!= inactive INACTIVE Exadata Storage Server.. 99 Exadata Storage Server. Exadata Storage Server Exadata Storage Server. Exadata Storage Server. Exadata Storage Server Oracle Clusterware. Exadata Storage Server Oracle ASM. Exadata Storage Server. Exadata Storage Server. # shutdown -h -y now LDom SPARC T4-4. LDom, OS. LDom, ( ).,. ( ). 99

1 2 3 I/O,. # ldm stop domainname LDom domainname stopped # ldm unbind-domain domainname I/O,. # ldm stop activedomainname LDom activedomainname stopped # ldm unbind-domain activedomainname. # shutdown -i5 -g0 -y. SPARC SuperCluster T4-4.... EPO SPARC SuperCluster T4-4. PDU. Oracle. 98 1 2 3.. Exadata Storage Server, SPARC T4-4, Sun ZFS Storage 7320. Sun ZFS Storage 7320. SPARCT4-4. 100 SPARC SuperClusterT4-4 2012 12

4 Exadata Storage Server. Oracle Solaris 11 Oracle Solaris OS. beadm(1m). SPARC SuperCluster.. SPARC Supercluster. Oracle Solaris 11. http://docs.oracle.com/cd/e23824_01/html/e21801/toc.html 101 102 103 104 104.. Oracle Solaris beadm.., beadm create..,,. beadm mount. pkg (-R). beadm create.,. 101

. # beadm create BeName@snapshotNamedescription BeName@snapshotdescription. BeName. snapshotdescription. beadm list -s. beadm create -e. beadm activate. Solaris 11. http://docs.oracle.com/cd/e23824_01/html/e21801/snap3.html#scrolltoc 1 beadm. SPARC.. localsys% ssh systemname -l root Password: Last login: Wed Nov 13 20:27:29 2011 from dhcp-vpn-r Oracle Corporation SunOS 5.11 solaris April 2011 root@sup46:~# 2 ZFS beadm. root@sup46:~# beadm list BE Active Mountpoint Space Policy Created ---------------------------------------------------------- solaris NR / 2.17G static 2011-07-13 12:01 Active. N (Now) R (Reboot). 102 SPARC SuperClusterT4-4 2012 12

3 ZFS. root@sup46:~# beadm create solaris_backup root@sup46:~# beadm list BE Active Mountpoint Space Policy Created ----------------------------------------------------------------- solaris NR / 2.17G static 2011-07-13 12:01 solaris_backup - - 35.0K static 2011-07-17 21:01 4. root@sup46:~# beadm activate solaris_backup root@sup46:~# beadm list BE Active Mountpoint Space Policy Created ----------------------------------------------------------------- solaris_backup R - 2.17G static 2011-07-17 21:01 solaris N / 1.86G static 2011-07-13 12:01 5. root@sup46:~# reboot Connection to systemname closed by remote host. Connection to systemname closed. localsys% ssh systemname -l root Password: Last login: Thu Jul 14 14:37:34 2011 from dhcp-vpn- Oracle Corporation SunOS 5.11 solaris April 2011 root@sup46:~# beadm list BE Active Mountpoint Space Policy Created ----------------------------------------------------------------- solaris_backup NR - 2.19G static 2011-07-17 21:01 solaris - / 4.12G static 2011-07-13 12:01. root@sup46:~# beadm mount solaris_backup /mnt root@sup46:~# df -k /mnt Filesystem 1024-blocks Used Available Capacity Mounted on rpool1/root/solaris_backup 286949376 2195449 232785749 1% /mnt root@sup46:~# df -k / Filesystem 1024-blocks Used Available Capacity Mounted on rpool1/root/solaris_backup 286949376 2214203 232785749 1% / root@sup46:~# ls /mnt bin etc lib opt rpool1 system wwss boot export media pkg sbin tmp 103

cdrom home micro platform scde usr dev import mnt proc share var devices java net re shared workspace doe kernel nfs4 root src ws root@sup46:~# root@sup46:~# beadm umount solaris root@sup46:~#. root@sup46:~# beadm activate solaris root@sup46:~# reboot Connection to systemname closed by remote host. Connection to systemname closed. localsys% ssh systemname -l root Password: Last login: Thu Jul 14 14:37:34 2011 from dhcp-vpn- Oracle Corporation SunOS 5.11 solaris April 2011 root@sup46:~#. root@sup46:~# beadm list BE Active Mountpoint Space Policy Created ----------------------------------------------------------------- solaris_backup - - 13.25G static 2011-07-17 21:19 solaris - - 4.12G static 2011-07-13 12:01 root@sup46:~# beadm destroy solaris_backup Are you sure you want to destroy solaris_backup? This action cannot be undone(y/[n]): y root@sup46:~# beadm list BE Active Mountpoint Space Policy Created ----------------------------------------------------------------- solaris NR / 4.12G static 2011-07-13 12:01 root@sup46:~# 104 SPARC SuperClusterT4-4 2012 12

DISM Dynamic Intimate Shared Memory 105 DISM 105 DISM DISM DISM(Dynamic Intimate Shared Memory) ASM SPARC SuperCluster Solaris.ASM SPARC SuperCluster DISM ( ). ASM. Solaris 11 Automatic Memory Management. Solaris 11 DISM DISM. 105 DISM. DISM Oracle DISM Oracle Dynamic SGA Tuning of Oracle Database on Oracle Solaris with DISM. http://www.oracle.com/ technetwork/articles/systems-hardware-architecture/ using-dynamic-intimate-memory-sparc-168402.pdf DISM DISM(Dynamic Intimate Shared Memory) ASM SPARC SuperCluster Solaris. 105 DISM. DBA ASMM(Automatic Shared Memory Management). Solaris DISM. SGA_MAX_SIZE / MEMORY_MAX_TARGET / MEMORY_TARGET, SGA_MAX_SIZE SGA_TARGET SGA., 64G SGA. alter system set SGA_TARGET=64G scope=spfile; alter system set SGA_MAX_SIZE=64G scope=spfile; 105

alter system set MEMORY_MAX_TARGET=0 scope=spfile; alter system set MEMORY_TARGET=0 scope=spfile; SPARC SuperCluster T4-4,.. 12 16 SPARC T4-4 http://download.oracle.com/docs/cd/e23411_01/index.html 19 Sun ZFS Storage 7320 19 Sun Datacenter InfiniBand Switch 36 20 Cisco Catalyst 4948 http://download.oracle.com/docs/cd/e22471_01/html/820-4167/toc.html http://download.oracle.com/docs/cd/e19197-01/index.html http://www.cisco.com/ en/us/docs/switches/lan/catalyst4900/4948e/installation/guide/ 4948E_ins.html 18 Exadata Storage Server 106 Exadata Storage Server Oracle VM Server for SPARC Oracle Solaris Cluster Oracle Solaris 11 Oracle Solaris 10 http://download.oracle.com/docs/cd/e23120_01/index.html http://download.oracle.com/docs/cd/e18728_01/index.html http://download.oracle.com/docs/cd/e19963-01/index.html http://download.oracle.com/docs/cd/e23823_01/index.html Exadata Storage Server Exadata Storage Server Oracle Exadata Smart Flash Cache Oracle I/O. 18 Exadata Storage Server. SPARC SuperCluster T4-4 Exadata Storage Server Exadata Storage Server. /opt/oracle/cell/doc 106 SPARC SuperClusterT4-4 2012 12

SPARC SuperCluster T4-4 Exadata Storage Server. 107 (Write-Through) 108 Exadata Storage Server 110 Exadata Storage Server Oracle ASM Oracle Exadata Storage Server Software User's Guide. 1 2 (Write-Through) Exadata Storage Server. (Write-Back) (Write-Through). (Write-Through) (Write-Back). Exadata Storage Server (Write-Back). 11.2.1.3 Exadata Storage Server. Oracle Exadata Storage Server 11.2.1.3 ( :1,4,7 10 17 1 )... CellCLI> ALTER CELL bbulearncycletime="2011-01-22t02:00:00-08:00". CellCLI> LIST CELL ATTRIBUTES bbulearncycletime Exadata Storage Server.. HDD disk controller battery on disk contoller at adapter 0 is going into a learn cycle. This is a normal maintena 3. # /opt/megaraid/megacli/megacli64 -AdpBbuCmd -GetBbuStatus -a0. BBU status for Adapter: 0 BatteryType: ibbu08 Voltage: 3721 mv Current: 541 ma 107

Temperature: 43 C BBU Firmware Status: Charging Status : Charging Voltage : OK Temperature : OK Learn Cycle Requested : No Learn Cycle Active : No Learn Cycle Status : OK Learn Cycle Timeout : No I2c Errors Detected : No Battery Pack Missing : No Battery Replacement required : No Remaining Capacity Low : Yes Periodic Learn Required : No Transparent Learn : No Battery state: GasGuageStatus: Fully Discharged : No Fully Charged : No Discharging : No Initialized : No Remaining Time Alarm : Yes Remaining Capacity Alarm: No Discharge Terminated : No Over Temperature : No Charging Terminated : No Over Charged : No Relative State of Charge: 7 % Charger System State: 1 Charger System Ctrl: 0 Charging current: 541 ma Absolute State of Charge: 0% Max Error: 0 % Exit Code: 0x00 Exadata Storage Server Exadata Storage Server. Exadata Storage Server Exadata Storage Server Oracle ASM. Exadata Storage Server Oracle ASM Exadata Storage Server Exadata Storage Server. Exadata Storage Server. 108 SPARC SuperClusterT4-4 2012 12

1 2 3 4 5 6 7. CellCLI> LIST GRIDDISK ATTRIBUTES name WHERE asmdeactivationoutcome!= Yes Oracle ASM Exadata Storage Server. Exadata Storage Server Oracle ASM. Exadata Storage Server. CellCLI> ALTER GRIDDISK ALL INACTIVE. Exadata Storage Server INACTIVE. CellCLI> LIST GRIDDISK WHERE STATUS!= inactive INACTIVE Exadata Storage Server..... CellCLI> ALTER GRIDDISK ALL ACTIVE Oracle ASM.. CellCLI> LIST GRIDDISK ATTRIBUTES name, asmmodestatus asmmodestatus ONLINE UNUSED.. DATA_CD_00_dm01cel01 DATA_CD_01_dm01cel01 DATA_CD_02_dm01cel01 DATA_CD_02_dm02cel01 DATA_CD_02_dm03cel01 DATA_CD_02_dm04cel01 DATA_CD_02_dm05cel01 DATA_CD_02_dm06cel01 DATA_CD_02_dm07cel01 DATA_CD_02_dm08cel01 ONLINE SYNCING OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE OFFLINE 109

DATA_CD_02_dm09cel01 OFFLINE DATA_CD_02_dm10cel01 OFFLINE DATA_CD_02_dm11cel01 OFFLINE Oracle ASM asmmodestatus=online asmmodestatus=unused. Exadata Storage Server Oracle ASM Exadata Storage Server. Exadata Storage Server.. CellCLI> list griddisk attributes name where asmdeactivationoutcome!= Yes DATA_CD_00_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_01_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_02_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_03_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_04_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_05_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_06_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_07_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_08_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_09_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_10_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" DATA_CD_11_dm01cel02 "Cannot de-activate due to other offline disks in the diskgroup" 1 2 3 4 5 Exadata Storage Server Oracle ASM Oracle ASM. ALTER DISKGROUP diskgroup-name DROP DISK asm-disk-name Oracle ASM. Exadata Storage Server cellip.ora IP. Exadata Storage Server,. DROP CELLDISK celldisk-on-this-lun FORCE Exadata Storage Server.. 108 Exadata Storage Server. 110 SPARC SuperClusterT4-4 2012 12

SPARC SuperCluster ssctuner Oracle Solaris 10/11 Perl/Korn. SMF. /etc/system ndd parameters /kernel/drv/sd.conf /kernel/drv/ibd.conf DISM NFS. 2. ssctuner, ssctuner. 2 zpool. ssctuner ssctuner.. 111 ssctuner 112 ssctuner 113 ssctuner Oracle Solaris OS SMF Oracle Solaris System Administration Guide: Common System Management Tasks. http://docs.oracle.com/ cd/e23824_01/html/821-1451/hbrunlevels-25516.html#scrolltoc ssctuner ssctuner. # svcs ssctuner 111

, ssctuner syslog ssctuner. ssctuner. ssctuner. # svcs -x ssctuner shows svc:/site/application/sysadmin/ssctuner:default (ssctuner for SPARC SuperCluster) State: online since September 28, 2012 07:30:15 AM PDT See: ssctuner(l) See: /var/svc/log/site-application-sysadmin-ssctuner:default.log Impact: None. # more /var/svc/log/site-application-sysadmin-ssctuner\:default.log [ Sep 28 07:30:00 Disabled. ] [ Sep 28 07:30:00 Rereading configuration. ] [ Sep 28 07:30:10 Enabled. ] [ Sep 28 07:30:10 Executing start method ("/opt/oracle.supercluster/ssctuner.ksh start"). ] ssctuner local0.notice success: Saved rollback for : /etc/system ssctuner local0.notice success: Saved ndd rollback. ssctuner local0.notice success: Saved rollback for : /kernel/drv/sd.conf ssctuner local0.notice success: enabled, version 0.99e. daemon PID= 14599 [ Sep 28 07:30:15 Method "start" exited with status 0. ] ssctuner local0.notice success: daemon executing ssctuner local0.notice success: Changes made to /etc/system ssctuner local0.notice success: Changes made to /kernel/drv/sd.conf /var/adm ssctuner. # grep -i ssctuner /var/adm/messages Sep 28 07:30:10 etc6cn04 ssctuner: [ID 702911 local0.notice] success: Saved rollback for : /etc/system Sep 28 07:30:10 etc6cn04 ssctuner: [ID 702911 local0.notice] success: Saved ndd rollback. Sep 28 07:30:10 etc6cn04 ssctuner: [ID 702911 local0.notice] success: Saved rollback for : /kernel/drv/sd.conf Sep 28 07:30:15 etc6cn04 ssctuner: [ID 702911 local0.notice] success: enabled, version 0.99e. daemon PID= 14599 Sep 28 07:30:15 etc6cn04 ssctuner: [ID 702911 local0.notice] success: daemon executing Sep 28 07:30:15 etc6cn04 ssctuner: [ID 702911 local0.notice] success: Changes made to /etc/system Sep 28 07:30:15 etc6cn04 ssctuner: [ID 702911 local0.notice] success: Changes made to /kernel/drv/sd.conf 113 ssctuner 112 ssctuner ssctuner ndd ssctuner. ssctuner. ssctuner 113 ssctuner.,. 112 SPARC SuperClusterT4-4 2012 12

CPU ssctuner. # svcadm disable ssctuner ssctuner ndd. 113 ssctuner 111 ssctuner 1 2 ssctuner ssctuner. SMF. /opt/oracle.supercluster/ssctuner/etc/ssctuner.conf FALSE. NDD_TUNE=TRUE SYSTEM_TUNE=TRUE SDCONF_TUNE=TRUE ZPOOL_FIX=TRUE NFS_CHECK=TRUE DISM_CHECK=TRUE SMF. 111 ssctuner 112 ssctuner CPU 114 CPU/Memory 114 CPU/ 115 CPU/ 120 CPU/ 113

CPU/Memory CPU/Memory SPARC SuperCluster T4-4 CPU. CPU/Memory (setcoremem). CPU ( ). 4 32GB, 4 32. 32 1TB.... CPU/Memory. Oracle Solaris 11.1. http://docs.oracle.com/cd/e26502_01/html/e28984/gihhp.html#gikvp 115 CPU/ 114 CPU/ 120 CPU/ CPU/ CPU,. CPU/Memory CPU/,. 13 1 2 3 (%) 0 1, 32Gb 2, 1 2 1 25% 75% 2, 1 2 2 50% 50% 2, 1 2 3 75% 25% 2, 1 2 114 SPARC SuperClusterT4-4 2012 12

CPU/ 13 ( ) 1 2 3 (%) 4 25% 25% 50% 2, 1 1, 2 1 5 25% 50% 25% 2, 1 1, 2 1 6 50% 25% 25% 2, 1 1, 2 1 7 25% 25% 25% 25% 1 128 100% 4 115 CPU/ 120 CPU/ 114 CPU/Memory CPU/ CPU... 1 2 3 root. ldm bind.. setcoremem CPU. setcoremem 2. setcoremem -s 0 CPU. setcoremem -s 1. 115

CPU/ setcoremem -s 0 ( 3) setcoremem -s 1 ( 5).. $ ssh root@etc8cn06-app1 Password: Last login: Mon Jul 11 1:03:50 2012 Oracle Corporation SunOS 5.11 11.0 May 2012 root@etc8cn06-app1:~# cd /opt/oracle.supercluster/bin root@etc8cn06-app1:/opt/oracle.supercluster/bin/setcoremem/lib/bin#./setcoremem -s 0 Info: A log of the activties is being recorded in /opt/oracle.supercluster/setcoremem/log/modlocality.log A previous configuration that has not been fully applied already exist, Do you want to reconfigure again? (y,n default: n) y Info: The current or Next Power On SP-Config is layout7_ml10212012031137 Info: Found 4 Domains including primary DOMAIN CORES Memory(GB) ====== ===== ========== primary 20 928 s11-sru2a-dump-1 4 32 s11-sru2a-dump-2 4 32 s11-sru2a-dump-3 4 32 With this configuration adjustment tool, existing domains can be re-configured with new CPU-core count and memory size. The minimum number of Cores for a domain is 4. Adjustments are allowed in multiple of 4 cores. The minimum amount of Memory for a domain is 32 GB, Adjustments are allowed in multiple of 32 GB. Info: Parking cores (for license purposes) ==================================== Parked cores are cores that are not assigned to any domain and are therefore not counted for license purposes. Parked cores are unusable and essentially wasted. A better alternative in most situations is to avoid unwanted license charges by assigning cores to a different domain. This resource adjustment tool cannot be used to modify the number of domains. Do you want to reconfigure cpu and memory for all 4 domains? (y,n default: n) y DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/0 928/0 s11-sru2a-dump-1 4/0 32/0 s11-sru2a-dump-2 4/0 32/0 s11-sru2a-dump-3 4/0 32/0 CPU. 0. Enter 0 as the value for Core or Memory if you wish to abort Valid Core Options are 0, 4, 8, 12, 16, 20 116 SPARC SuperClusterT4-4 2012 12

CPU/ For [primary ] Enter the number of Cores between 4-20: 4 Valid Memory Options are 0, 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800, 832, 864, 896, 928 For [primary ] Enter Memory size in GB between 32-928: 32 DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/4 928/32 s11-sru2a-dump-1 4/0 32/0 s11-sru2a-dump-2 4/0 32/0 s11-sru2a-dump-3 4/0 32/0 CPU/Memory 0. Enter 0 as the value for Core or Memory if you wish to abort Valid Core Options are 0, 4, 8, 12, 16, 20 For [s11-sru2a-dump-1 ] Enter the number of Cores between 4-20: 8 Valid Memory Options are 0, 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800, 832, 864, 896, 928 For [s11-sru2a-dump-1 ] Enter Memory size in GB between 32-928: 288 DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/4 928/32 s11-sru2a-dump-1 4/8 32/288 s11-sru2a-dump-2 4/0 32/0 s11-sru2a-dump-3 4/0 32/0 Enter 0 as the value for Core or Memory if you wish to abort Valid Core Options are 0, 4, 8, 12, 16 For [s11-sru2a-dump-2 ] Enter the number of Cores between 4-16: 8 Valid Memory Options are 0, 32, 64, 96, 128, 160, 192, 224, 256, 288, 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672 For [s11-sru2a-dump-2 ] Enter Memory size in GB between 32-672: 288 DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/4 928/32 s11-sru2a-dump-1 4/8 32/288 s11-sru2a-dump-2 4/8 32/288 s11-sru2a-dump-3 4/0 32/0 CPU. Enter 0 as the value for Core or Memory if you wish to abort Valid Core Options are 0, 4, 8, 12, 16 For [s11-sru2a-dump-3 ] Enter the number of Cores between 4-16: 8 Valid Memory Options are 0, 32, 64, 96, 128, 160, 192, 224, 256, 288, 117

CPU/ 320, 352, 384, 416, 448, 480, 512, 544, 576, 608, 640, 672 For [s11-sru2a-dump-3 ] Enter Memory size in GB between 32-672: 288 DOMAIN CORES-Cur/New Mem(GB)-Cur/New ====== ============= =============== primary 20/4 928/32 s11-sru2a-dump-1 4/8 32/288 s11-sru2a-dump-2 4/8 32/288 s11-sru2a-dump-3 4/0 32/288 [4 Cores] and [128 GB] of Memory will be parked Do you wish to continue with these changes? (y,n default: n) y Please Execute./setcoremem -s 1 and follow the instructions there Execution of above command will cause all domains in this system to be halted and will require a reboot of this domain. Please enusre that you have stopped all applications in all domains on this system, before proceeding with the above command root@etc8cn06-app1:/opt/oracle.supercluster/bin#,. 4 5, ( ). setcoremem -s 1. ( ). root@etc8cn06-app1:/opt/oracle.supercluster/bin#./setcoremem -s 1 -v Info: A log of the activties is being recorded in /opt/oracle.supercluster/log/modlocality.log Info: Executing step 1 - Please do not interrupt until this step is complete The system has 1024 GB memory Info: The current ldoms config is layout7_ml10212012031137 Info: It will remain preserved, and a new config will be added at end of step 2 Info: The name of the new config will be layout7_ml10222012031925 LDom s11-sru2a-dump-3 stopped Warning: Resulting memory size is zero. Domain s11-sru2a-dump-3 uses physically bound core resources Warning: setting required number of cores to zero LDom s11-sru2a-dump-2 stopped Warning: Resulting memory size is zero. Domain s11-sru2a-dump-2 uses physically bound core resources Warning: setting required number of cores to zero LDom s11-sru2a-dump-1 stopped Warning: Resulting memory size is zero. Domain s11-sru2a-dump-1 uses physically bound core resources Warning: setting required number of cores to zero Initiating a delayed reconfiguration operation on the primary domain. 118 SPARC SuperClusterT4-4 2012 12

CPU/ All configuration changes for other domains are disabled until the primary domain reboots, at which time the new configuration for the primary domain will also take effect. ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ Info: Adding 4 Cores to Primary Domain... ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Notice: The primary domain is in the process of a delayed reconfiguration. Any changes made to the primary domain will only take effect after it reboots. ------------------------------------------------------------------------------ Info: Done with changes Info: Done with step 1 - system reboot is required Info: After reboot, Please run setcoremem -s 2 root@etc8cn06-app1:/opt/oracle.supercluster/bin# 6 7. setcoremem -s 2. $ ssh root@etc8cn04-app1 Password: Last login: Mon Aug 23 10:44:50 2012 Oracle Corporation SunOS 5.11 11.0 May 2012 You have new mail. root@ssc6cn06:~# cd /opt/oracle.supercluster/bin root@etc8cn06-app1:/opt/oracle.supercluster/bin#./setcoremem -s 2 -v Info: A log of the activties is being recorded in /opt/oracle.supercluster/log/modlocality.log Info: Executing step 2 - Please do not interrupt until this step is complete Info: Setting CORE Constraints for Domain s11-sru2a-dump-1 Info: Setting COREs to 1,2,9,10,17,18,25,26 for Domain s11-sru2a-dump-1 Info: Setting Memory for Domain s11-sru2a-dump-1 Info: Binding Domain s11-sru2a-dump-1 Info: Starting Domain s11-sru2a-dump-1 LDom s11-sru2a-dump-1 started Info: Setting CORE Constraints for Domain s11-sru2a-dump-2 Info: Setting COREs to 3,4,11,12,19,20,27,28 for Domain s11-sru2a-dump-2 Info: Setting Memory for Domain s11-sru2a-dump-2 Info: Binding Domain s11-sru2a-dump-2 Cannot validate NIC net9 on service domain s11-sru2a-dump-2 Cannot validate path /dev/rdsk/c0t5000c500436febb7d0s1 on service domain s11-sru2a-dump-1 Info: Starting Domain s11-sru2a-dump-2 LDom s11-sru2a-dump-2 started Info: Setting CORE Constraints for Domain s11-sru2a-dump-3 Info: Setting COREs to 5,6,13,14,21,22,29,30 for Domain s11-sru2a-dump-3 119

CPU/ Info: Setting Memory for Domain s11-sru2a-dump-3 LDom s11-sru2a-dump-3 started Info: The new SP Config is layout7_ml10222012031925 Info: Done with changes Info: CPU and Memory configuration successfully changed root@etc8cn06-app1:/opt/oracle.supercluster/bin# 120 CPU/ 114 CPU/ 114 CPU/Memory CPU/ CPU/... CPU/. SPARC SuperCluster T4-4. 1.. /opt/oracle.supercluster/setcoremem/log $ ssh root@hostname-app1 Password: Last login: Mon Jul 11 1:53:50 2012 Oracle Corporation SunOS 5.11 11.0 May 2012 # ldm list-confg factory-default layout6 layout6_ml10182012142558 layout6_ml10182012162049 layout6_ml10182012165059 layout6_ml10182012201150 [current] layout6..ml Modified Layout( ) (MMDDYYYYHHMISS). layout6_ml10182012201150 2012 10 18 8 11 50. 2. # ldm set-config layout6 120 SPARC SuperClusterT4-4 2012 12

CPU/ 3 4.. # #. -> cd /SP -> stop /SYS Are you sure you want to stop /SYS (y/n)? y Stopping /SYS -> start /SYS Are you sure you want to start /SYS (y/n)? y Starting /SYS 5. 114 CPU/Memory 115 CPU/ 114 CPU/ 121

122

SPARC SuperClusterT4-4 SPARC SuperCluster T4-4. 123 SPARC SuperCluster T4-4 141 OCM SPARC SuperCluster T4-4 146 EM Exadata SPARC SuperCluster SPARC SuperClusterT4-4 Oracle (ASR) SPARC SuperCluster T4-4. 123 ASR 124 ASR 126 ASR Manager 128 ASR Manager 129 Exadata Storage Server SNMP 131 Sun ZFS Storage 7320 ASR 133 SPARC T4-4 ASR : Oracle ILOM 135 SPARC T4-4 ASR : Oracle Solaris 11 139 SPARC SuperCluster T4-4 ASR ASR (ASR) SPARC SuperCluster T4-4. SPARC SuperCluster T4-4 ASR Manager. ASR Manager SPARC SuperCluster T4-4 HTTPS HTTPS. ASR Manager Oracle. Oracle /. 123

ASR ASR. Oracle Premier Support Oracle/Sun SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 MOS(My Oracle Support).. ASR. ASR.,.. http://www.oracle.com/asr Documentation ASR Fault Coverage Information. ASR SMTP SNMP.. ASR. Oracle.. SNMP ASR Manager. Oracle. ASR Oracle (http:// www.oracle.com/asr). ASR SPARC SuperCluster T4-4 ASR. 125 ASR 125 : ASR Manager 125 : SPARC SuperCluster T4-4 124 SPARC SuperClusterT4-4 2012 12

: SPARC SuperClusterT4-4 1 2 3 4 5 ASR ASR. http://support.oracle.com MOS(My Oracle Support).. Oracle Premier Support Oracle/Sun SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 ASR Manager.. http://www.oracle.com/asr additional details( ) Hardware and Network Configuration Recommendations( ). ASR. HTTPS.. Oracle ASR Security Whitepaper. http://www.oracle.com/technetwork/systems/asr/documentation/index.html. SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 : ASR Manager ASR Manager. ASR Manager, 3.6 Solaris Oracle STB(Services Tool Bundle) : SPARC SuperClusterT4-4 ASR Manager. : Oracle Linux 5.3 Oracle Solaris 10 Update 10(10u10) + Java : JRE/JDK 6.2.28 : Exadata 11.2.3.1 125

ASR Manager Exadata Storage Server: Exadata Storage Server 11.2.0.3 DB 1 Sun ZFS Storage 7320 : AK2011.1.0 ASR Manager ASR Manager ASR Manager. 125 ASR ASR Manager. 126 OASM(Oracle Automated Service Manager) 126 Solaris STB(Service Tools Bundle) 127 Oracle ASR( ) 127 ASR Manager 1 2 OASM(Oracle Automated Service Manager) 1.3.1 ( OASM ). : Oracle Solaris: pkginfo -l SUNWsasm Oracle Linux: rpm -q SUNWsasm OASM. : Oracle Solaris: pkgadd -d SUNWsasm.version-number.pkg Oracle Linux: rpm -i SUNWsasm.version-number.rpm 1 2 Solaris STB(ServiceTools Bundle) ServicesTools Bundle. http://www.oracle.com/asr Download( ). STB (install_stb.sh).. "install" I 126 SPARC SuperClusterT4-4 2012 12

ASR Manager SNEEP Y Y Doc ID 1153444.1 My Oracle Support STB ( ). https://support.oracle.com 3 4 SNEEP. sneep -a. stclient -E. sneep -s serial-number 1 2 Oracle ASR( ) ASR. : Oracle Solaris: pkgadd -d SUNWswasr.version-number.pkg Oracle Linux: rpm -i SUNWswasr.version-number.rpm asr PATH..profile,.cshrc,.kshrc.bashrc. PATH=$PATH:/opt/SUNWswasr/bin export PATH 1 2 ASR Manager ASR Manager, MOS. ASR Manager. # asr register "1" "alternate URL for Managed OPS use only". 1) transport.oracle.com 127

ASR Manager 3.. Proxy server name:? <proxy server name> Proxy port number:? <proxy port number> Proxy authentication; if authentication is not required, enter -. Proxy user: <proxy user name> Proxy password: <proxy password> If this is an NTLM type proxy, enter the information below. Otherwise, enter - NTLM Domain: [?] <NTLM domain name> Enter the host the NTLM authentication request is originating from. Usually this is the hostname of the SASM server. NTLM Host: [?] <NTLM host name> NTLM support is not bundled with SASM but may be added now. 1) Download jcifs from http://jcifs.samba.org/ 2) Extract contents and locate the jcifs-*.jar file 3) Enter full path to this file below jcifs jar file: [?] <full path of jcifs jar file> Note: These properties are stored in the /var/opt/sunwsasm/configuration/config.ini file. You can update these properties if needed and then restart SASM. 4 MOS(My Oracle Support). ASR.. :. MOS ASR (Service Request, SR). 1 2 3 ASR Manager ASR Manager ASR Manager. # asr show_rules_version 3.6.. # asr show_reg_status ASR. # asr test_connection 128 SPARC SuperClusterT4-4 2012 12

ASR Manager Exadata Storage Server SNMP.. Exadata Storage Server. 1 2 Exadata Storage Server celladmin. Exadata Storage Server SNMP. # cellcli -e alter cell snmpsubscriber=(host = ASR-Manager-name-or-IP-address,port=162,community=public,type=asr) ASR-Manager-name-or-IP-address.. host='asr-manager-name-or-ip-address' ASR Manager DNS.DNS IP, /etc/hosts ASR Manager. type=asr ASR Manager SNMP. community=public.. port=162 SNMP.., ASR. 3 Oracle ILOM ( Oracle ILOM ). # asr list_asset. IP_ADDRESS HOST_NAME SERIAL_NUMBER ASR PROTOCOL SOURCE ---------- --------- ------------- --- -------- ----- 10.60.40.105 ssc1cel01 1234FMM0CA Enabled SNMP ILOM 10.60.40.106 ssc1cel02 1235FMM0CA Enabled SNMP ILOM 10.60.40.107 ssc1cel03 1236FMM0CA Enabled SNMP ILOM 10.60.40.117 ssc1cel01-ilom 1234FMM0CA Enabled SNMP,HTTP EXADATA-SW 10.60.40.118 ssc1cel02-ilom 1235FMM0CA Enabled SNMP,HTTP EXADATA-SW 10.60.40.119 ssc1cel03-ilom 1236FMM0CA Enabled SNMP,HTTP EXADATA-SW Exadata Storage Server Oracle ILOM 5. Oracle ILOM 4. 129

ASR Manager 4 ASR Manager Exadata Storage Server Oracle ILOM. # asr activate_asset -i ILOM-IP-address # asr activate_asset -h ILOM-hostname Oracle ILOM 6481. 6481 ASR. 5 ASR Exadata OS. # asr activate_exadata -i host-management-ip-address -h host-management-hostname -l ILOM-IP-address # asr activate_exadata -i host-management-ip-address -h host-management-hostname -n ILOM-hostname 6 ASR Manager Exadata Storage Server. # asr list_asset Oracle ILOM. IP_ADDRESS HOST_NAME SERIAL_NUMBER ASR PROTOCOL SOURCE ---------- --------- ------------- --- -------- ----- 10.60.40.105 ssc1cel01 1234FMM0CA Enabled SNMP ILOM 10.60.40.106 ssc1cel02 1235FMM0CA Enabled SNMP ILOM 10.60.40.107 ssc1cel03 1236FMM0CA Enabled SNMP ILOM 10.60.40.117 ssc1cel01-ilom 1234FMM0CA Enabled SNMP,HTTP EXADATA-SW 10.60.40.118 ssc1cel02-ilom 1235FMM0CA Enabled SNMP,HTTP EXADATA-SW 10.60.40.119 ssc1cel03-ilom 1236FMM0CA Enabled SNMP,HTTP EXADATA-SW 7 8 Exadata Storage Server. # cellcli -e list cell attributes snmpsubscriber Exadata Storage Server SNMP. # cellcli -e alter cell validate snmp type=asr MOS. 9 10 SPARC SuperClusterT4-4 Exadata Storage Server. SPARC SuperClusterT4-4 Exadata Storage Server MOS Exadata Storage Server. 139 SPARC SuperCluster T4-4 ASR. 130 SPARC SuperClusterT4-4 2012 12

ASR Manager ASR MOS 5.3+ (Doc ID 1329200.1). Sun ZFS Storage 7320 ASR SPARC SuperCluster T4-4 Sun ZFS Storage 7320 Sun ZFS Storage 7320. 1 2 3 Sun ZFS Storage 7320 IP. https://storage-controller-ipaddress:215 https://storage-controller-hostname:215. Username( ) root root Enter. Configuration( ) SERVICES( ) Services( ). 131

ASR Manager 4 Phone Home( ). Phone Home( ) Phone Home( ). 5 Use web proxy( ). 132 SPARC SuperClusterT4-4 2012 12

ASR Manager Host:port( : ). Username( ). Password( ). 6 7 8 9 10 11. Privacy Statement( ).OK( ) My Oracle Support OK( ). Sun Inventory and Enable Phone Home(Sun Inventory ). APPLY( ). Service Enable / Disable( / ) Enable( ). SPARC SuperClusterT4-4 Sun ZFS Storage 7320. SPARC SuperClusterT4-4 Sun ZFS Storage 7320 MOS Sun ZFS Storage 7320. 139 SPARC SuperCluster T4-4 ASR. ASR MOS 5.3+ (Doc ID 1329200.1). SPARCT4-4 ASR : Oracle ILOM.. SPARC T4-4 Oracle ILOM SPARC T4-4. 1 SPARCT4-4 Oracle ILOM. 133

ASR Manager 2. # show /SP/alertmgmt/rules. 1 2 3... 15 3. # show /SP/alertmgmt/rules/rule-number. # show /SP/alertmgmt/rules/1 Properties: type = snmptrap level = minor destination = 10.60.10.243 destination_port = 0 community_or_username = public snmp_version = 2c testrule = (Cannot show property). ASR Manager IP. show /SP/alertmgmt/rules/rule-number.. Properties: type = snmptrap level = disable destination = 0.0.0.0 destination_port = 0 community_or_username = public 134 SPARC SuperClusterT4-4 2012 12

SPARCT4-4 ASR : Oracle Solaris 11 snmp_version = 1 testrule = (Cannot show property). 4 5 6 7 8. # set /SP/alertmgmt/rules/unused-rule-number type=snmptrap level=minor destination=ip-address-of-asr-manager snmp_version=2c community_or_username=public ASR Manager. SPARCT4-4 Oracle ILOM. asr> activate_asset -i ILOM-IP-address SPARC SuperClusterT4-4 SPARCT4-4 Oracle ILOM. SPARC SuperClusterT4-4 SPARCT4-4 MOS SPARCT4-4. 139 SPARC SuperCluster T4-4 ASR. ASR MOS 5.3+ (Doc ID 1329200.1). SPARCT4-4 ASR : Oracle Solaris 11.. Oracle Solaris 11 ASR Manager HTTP xml ASR Oracle. ASR Manager asr enable_http_receiver. HTTP.. 136 ASR Manager HTTP 136 ASR Manager HTTPS ( ) 135

SPARCT4-4 ASR : Oracle Solaris 11 137 ASR Manager Oracle Solaris 11 SPARC T4-4 ASR Manager HTTP Oracle Solaris 11 ASR HTTP ASR Manager. 1 2 3 root ASR Manager.. # asr show_http_receiver HTTP. # asr enable_http_receiver -p port-number port-number HTTP. HTTP asr disable_http_receiver. 4 5. # asr show_http_receiver HTTP. http://asr-manager-name:port-number/asr. HTTP. ASR Manager HTTPS ( ) HTTPS ASR Manager HTTP HTTPS/SSL. Jetty HTTPs/SSL. http://docs.codehaus.org/display/jetty/how+to+configure+ssl 1 SSL /var/opt/sunwsasm/configuration/jetty/jetty.xml <Call name="addconnector"> SSL. <Call name= addconnector > <Arg> <New class= org.mortbay.jetty.security.sslsocketconnector > 136 SPARC SuperClusterT4-4 2012 12

SPARCT4-4 ASR : Oracle Solaris 11 <Set name= Port >443</Set> <Set name= maxidletime >30000</Set> <Set name= keystore >path-to-keystore</set> <Set name= password >password</set> <Set name= keypassword key-password</set> <Set name= truststore >path-to-keystore</set> <Set name= trustpassword trust-password</set> </New> </Arg> </Call>. java -classpath lib/jetty-6.1.7.jar:lib/jetty-util-6.1.7.jar org.mortbay.jetty.security.password plaintext-password OBF:(OBF: ) jetty.xml. 2 OASM. Oracle Solaris. # svcadm restart sasm Oracle Linux. # /opt/sunwsasm/bin/sasm stop-instance # /opt/sunwsasm/bin/sasm start-instance 3 URL SSL. https://asr-manager-name/asr ASR Manager Oracle Solaris 11 SPARCT4-4 ASR Manager Oracle Solaris 11 SPARC T4-4. 1 2 SPARCT4-4 root. asr-notify. # svcs asr-notify : svcs: Pattern asr-notify doesn t match any instances asr-notify. # pkg list asr-notify 137

SPARCT4-4 ASR : Oracle Solaris 11 : pkg list: no packages matching asr-modify installed asr-notify. # pkg install system/fault-management/asr-notify svcs asr-notify asr-notify. : # svcs asr-notify STATE STIME FMRI online 16:06:05 svc:/system/fm/asr-notify:default asr-notify. 3 ASR Manager. # asradm register -e http://asr-manager-host:port-number/asr. # asradm register -e http://asrmanager1.mycompany.com:8777/asr Oracle Support. Oracle Support. Enter Orcle SSO User Name: Enter password: Registration complete. 4 5. # asradm list. PROPERTY VALUE Status Successfully Registered with ASR manager System Id system-identification-number Asset Id asset-identification-number User username Endpoint URL http://asr-manager-host:port-number/asr ASR Manager. SPARC SuperClusterT4-4 Oracle Solaris 11 SPARCT4-4. 138 SPARC SuperClusterT4-4 2012 12

SPARCT4-4 ASR : Oracle Solaris 11 6 SPARC SuperClusterT4-4 SPARCT4-4 MOS SPARCT4-4. 139 SPARC SuperClusterT4-4 ASR. ASR MOS 5.3+ (Doc ID 1329200.1). 1 2 3 4 SPARC SuperClusterT4-4 ASR ASR Manager SPARC SuperClusterT4-4. list_asset SPARC T4-4, Exadata Storage Server Sun ZFS Storage 7320 SPARC SuperCluster T4-4 ASR. My Oracle Support(https://support.oracle.com). My Oracle Support Dashboard( ) More( )... Settings( ). Settings( ) Administrative( ) Pending ASR Activations( ASR ). ASR. 139

SPARCT4-4 ASR : Oracle Solaris 11... SPARC SuperCluster T4-4. Oracle ILOM ASR. ASR. "Problem Category( )" "My - Auto Service Request (ASR) Installation and Configuration Issues( - (ASR) )" SR. 5..ASR Activation(ASR ). ASR Oracle ASR.( : ASR Manager asr activate_asset ) ASR. Contact Name( ):.. "SR ". Street Address 1( 1):. 140 SPARC SuperClusterT4-4 2012 12

OCM... Country( ):. ZIP/Postal Code( ):. "-". Distribution Email List( ): ASR... asr-notifications-1@mycompany.com, asr-notifications-2@mycompany.com ASR. ASR. 6 ASR "Approve( )". SPARC SuperCluster T4-4 My Oracle Support ASR. 7 ASR. # asradm send test email-address@company.com. OCM SPARC SuperClusterT4-4 OCM. 141 OCM 142 SPARC T4-4 Oracle Configuration Manager OCM Oracle Configuration Manager (OCM) Oracle. Oracle.. Oracle Configuration Manager. 141

OCM Oracle Oracle Configuration Manager ORACLE_HOME. Oracle Configuration Manager.. Oracle Configuration Manager Oracle.. Oracle Configuration Manager Installation and Administration Guide Oracle Configuration Manager Collection Overview SPARCT4-4 Oracle Configuration Manager.. 1 2 root. IP 1GbE.. ssh -l root ssc0101-mgmt SPARC T4-4 IP 1GbE.. ssh -l root ssc01db01 Oracle Configuration Manager. Oracle Configuration Manager. /opt/ocm Oracle Configuration Manager. /usr/lib/ocm 142 SPARC SuperClusterT4-4 2012 12

OCM Oracle Configuration Manager Oracle Configuration Manager. Oracle Configuration Manager ocm-install-dir. 3 4 5 6. /ocm-install-dir/ccr/bin emccr. emccr Oracle Configuration Manager SPARC T4-4.. emccr Oracle Configuration Manager SPARC T4-4.. SPARCT4-4 Oracle Configuration Manager. /ocm-install-dir/ccr/bin/configccr Oracle Configuration Manager. / ( ) SPARC T4-4 Oracle Configuration Manager. 7. /ocm-install-dir/ccr/admin/scripts/installccrsql.sh collectconfig -s $SID -r SYS XML /ocm-install-dir/ccr/hosts/sparc-t4-4-server/state/review. XML. XML Oracle. 8 9. ORACLE_HOME/ccr/bin/emCCR collect My Oracle Support. 143

OCM 10 Dashboard( ) Customize( ). Targets( ). 144 SPARC SuperClusterT4-4 2012 12

OCM 11. 12. 145

EM Exadata SPARC SuperCluster EM Exadata SPARC SuperCluster SPARC SuperCluster 1.1 Oracle Enterprise Manager Exadata 12.1.0.3 Exadata. Control LDOM 1.1 Oracle SuperCluster. SPARC SuperCluster October 2012 QMU. compmon pkg. pkg info compmon pkg list compmon. compmon. pkg://exa-family/system/platform/exadata/compmon@0.5.11,5.11-0.1.0.11:20120726t024158z : Exadata 12.1.0.3 exadatadiscoveryprecheck.pl catalog.xml. Discovery Precheck Script My Oracle Support exadatadiscoveryprecheck.pl. Oracle Enterprise Manager Exadata Management Getting Started Guide (docs.oracle.com/cd/e24628_01/doc.121/e27442/title.htm) Enterprise Manager Exadata Storage DB. Exadata Storage DB Oracle Exadata Storage Server Oracle Database Exadata Storage Server System Exadata Storage.. Oracle SuperCluster perfquery 1.5.8, HCA Port Errors (ID 15919339).HCA Enterprise Manager.. perfquery -V 146 SPARC SuperClusterT4-4 2012 12

Exalogic SPARC SuperCluster T4-4 Exalogic. 147 Exalogic 147 Exalogic 148 149 152 Dept1_Cluster1 GridLink 156 Dept1_Cluster1 SDP JDBC 158 Exalogic SDP InfiniBand Exalogic Oracle EECS(Exalogic Elastic Cloud Software) /, SPARC SuperCluster T4-4. /., CPU.. WebLogic Server. WebLogic Server. Sockets Direct Protocol InfiniBand, TCP CPU. Exalogic SPARC SuperCluster T4-4 Exalogic. Oracle Exalogic Enterprise Deployment Guide Chapter 3, Network, Storage, and Database Preconfiguration (http://docs.oracle.com/cd/e18476_01/doc.220/e18479/ toc.htm),,,. 147

Exalogic Oracle Exalogic Oracle Exalogic Enterprise Deployment Guide Chapter 5, Configuration Oracle Fusion Middleware (http://docs.oracle.com/cd/e18476_01/ doc.220/e18479/toc.htm). 1 2 3 4 5. OracleWebLogic Server. Domainname. Domainname. General( ). Enable Exalogic Optimizations(Exalogic ) Save( )... Enable Exalogic Optimizations(Exalogic ).. MBean I/O. -Dweblogic.ScatteredReadsEnabled=true/false KernelMBean.setScatteredReadsEnabled MBean I/O. -Dweblogic.GatheredWritesEnabled=true/false KernelMBean.setGatheredWritesEnabled MBean. -Dweblogic.replication.enableLazyDeserialization=true/false ClusterMBean.setSessionLazyDeserializationEnabled 148 SPARC SuperClusterT4-4 2012 12

Exalogic. java.io.ioexception: Broken pipe... WebLogic. Coherence*web. Oracle Fusion Middleware Exalogic Enterprise Deployment Guide Chapter 8, Deploying a Sample Web Application to and Oracle WebLogic Cluster (http:// docs.oracle.com/cd/e18476_01/doc.220/e18479/deploy.htm) dizzyworld.ear. Dept1_Cluster1. 1 2 Oracle Fusion Middleware Exalogic Enterprise Deployment Guide Section 5.16 Starting Managed Servers on ComputeNode1 and ComputeNode2 (http://docs.oracle.com/cd/ E18476_01/doc.220/e18479/create_domain.htm#BABEGAFB), Dept1_Cluster1. ( : WLS1). a. Domain Structure( ) Environment( ) Servers( ). Summary of Servers( ). b. WLS1. Settings forwls1(wls1 ). c. Cluster( ). d. Replication Ports( ). Dept_1_Cluster1 7005-7015. 7005-7015. 3 ( : WLS1). a. OracleWebLogic Server. 149

Exalogic b. Lock & Edit( ). c. Environment( ) Servers( ). Summary of Servers( ). d. Servers( ) WLS1. e. Protocols( ), Channels( ). f. New( ). g. ReplicationChannel t3 Next( ). h.. : 10.0.0.1 WLS1 IP. : 7005 i. Next( ) Network Channel Properties( ) Enabled( ) Outbound Enabled( ). j. Finish( ). k. Network Channels( ) WLS1 ReplicationChannel. l. Advanced( ) Enable SDP Protocol(SDP ). m. Save( ). n. Activate Changes( ). Dept1_Cluster1.. Dept1_Cluster1 WLS2 ReplicationChannel t3 10.0.0.2 7005 7006-7014 WLS3 ReplicationChannel t3 10.0.0.3 7005 7006-7014 150 SPARC SuperClusterT4-4 2012 12

Exalogic Dept1_Cluster1 WLS4 ReplicationChannel t3 10.0.0.4 7005 7006-7014 WLS5 ReplicationChannel t3 10.0.0.5 7005 7006-7014 WLS6 ReplicationChannel t3 10.0.0.6 7005 7006-7014 WLS7 ReplicationChannel t3 10.0.0.7 7005 7006-7014 WLS8 ReplicationChannel t3 10.0.0.8 7005 7006-7014 4 5 6 7 8 9 10 11 Environment( ) > Clusters( ). Summary of Clusters( ). Dept1_Cluster1( ). Settings for Dept1_Cluster1(Dept1_Cluster1 ). Replication( ). Replication Channel( ) ReplicationChannel. Advanced( ) Enable One Way RMI for Replication( RMI ). Save( ).. startweblogic.sh (base_domain bin ) -Djava.net.preferIPv4Stack=true. a. startweblogic.sh.. ${DOMAIN_HOME}/bin/setDomainEnv.sh $* b.. JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true" c.. 151

Dept1_Cluster1 GridLink 12. a. Environment( ) > Servers( ). Summary of Servers( ). b. WLS1 ( : WLS1). Settings forwls1(wls1 ). c. Control( ). Server Status( ) WLS1. Start( ). d. WebLogic. netstat -na. Dept1_Cluster1 GridLink Oracle WebLogic Server Oracle RAC JDBC GridLink. ONS(Oracle Notification Service) Oracle RAC.. 152 GridLink 154 Dept1_Cluster1 GridLink GridLink GridLink Oracle RAC. 152 153 153 XA 153 SCAN 153 Oracle Wallet GridLink. 152 SPARC SuperClusterT4-4 2012 12

OracleWallet / Oracle RAC.. Oracle RAC. Oracle RAC Oracle Database JDBC Developer's Guide and Reference Fast Connection Failover. http://docs.oracle.com/cd/b19306_01/java.102/b14355/ fstconfo.htm. WebLogic Server. CPU,, Oracle RAC FAN(Fast Application Notification), GridLink Oracle RAC. XA XA Oracle RAC Oracle RAC.XA RCLB. Oracle RAC. SCAN SCAN(Oracle Single Client Access Name) WebLogic TNS ONS. Oracle RAC SCAN GridLink. SCAN URL. http://www.oracle.com/technetwork/database/clustering/overview/scan-129069.pdf. OracleWallet Oracle Wallet ONS. 153

OracleWallet Dept1_Cluster1 GridLink Oracle GridLink.. XA. Dept1_Cluster1. 0. Oracle WebLogic Server Services( ), JDBC Datasources( ). Datasources( ) Datasource Name( ) Connection Pool( ) Initial capacity( ) 0. ONS.ONS onsctl. start GridLink. 1 2 3 4 5 6 7 OracleWebLogic Server. Lock & Edit( ). Domain Structure( ) Services( ) Data Sources( ). Summary of Data Sources( ) New( ) GridLink Data Source(GridLink ). Create a New JDBC GridLink Data Source( JDBC GridLink ).. Name( ) ( : gridlink). JNDI ( : jdbc/gridlink). Next( ). Transaction Options( ) Supports Global Transactions( ) Next( ). Enter individual listener information( ) Next( ). 154 SPARC SuperClusterT4-4 2012 12

OracleWallet 8. Service Name( ): Service Name( ) Oracle RAC. Service Name( ) myservice. Oracle RAC. Host Name( ) - DNS IP. Oracle GridLink -,. Port( ) -. Database User Name( ): ( : mydatabase). Password( ): ( : mypassword1). Next( ). Oracle Fusion Middleware Oracle WebLogic Server. JDBC URL.. jdbc:oracle:thin:@(description=(address_list=(address=(protocol=tcp)(host=left)(port=1 9 10 Test GridLink Database Connection(GridLink ) Test All Listeners( ). Oracle WebLogic... Next( ). ONS Client Configuration(ONS ). Fan Enabled(FAN ) Oracle FAN. ONS ONS FAN ONS. SCAN(Single Client Access Name) FAN. Next( ). 155

Dept1_Cluster1 SDP JDBC 11 12 13 14 15 Test ONS client configuration(ons ) Test All ONS Nodes( ONS ). Next( ). Select Targets( ) Dept1_Cluster1 All Servers in the cluster( ). Finish( ). Activate Changes( ). SDP JDBC. 156 Dept1_Cluster1 SDP JDBC. Dept1_Cluster1 SDP JDBC Dept1_Cluster1 SDP JDBC.. 156 Infiniband 156 JDBC SDP 157 Oracle Solaris netstat SDP Infiniband Oracle Database Net Services Administrator's Guide " Configuring SDP Protocol Support for Infinband Network Communication to the Database Server",JDBC SDP InfiniBand. http://download.oracle.com/ docs/cd/b28359_01/network.111/b28316/performance.htm#i1008413 SDP. 1 JDBC SDP. Oracle Fusion Middleware Exalogic Enterprise Deployment Guide Section 7.6 Configuring Grid Link Data Source for Dept1_Cluster1 (http://docs.oracle.com/cd/ 156 SPARC SuperClusterT4-4 2012 12

Infiniband E18476_01/doc.220/e18479/optimization.htm#BABHEDI), ComputeNode1 ComputeNode2 JDBC GridLink., JDBC URL. jdbc:oracle:thin:@(description=(address=(protocol=tcp)(host=192.x.x.x) (PORT=1522))(CONNECT_DATA=(SERVICE_NAME=myservice))) 2 3 JDBC URL TCP SDP.. jdbc:oracle:thin:@(description=(address=(protocol=sdp)(host=192.x.x.x) (PORT=1522))(CONNECT_DATA=(SERVICE_NAME=myservice))) startweblogic.sh (base_domain bin ) -Djava.net.preferIPv4Stack=true. a. startweblogic.sh.. ${DOMAIN_HOME}/bin/setDomainEnv.sh $* b.. JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true -Doracle.net.SDP=true" c.. 4. a. Environment( ) > Servers( ). Summary of Servers( ). b. WLS1 ( : WLS1). Settings forwls1(wls1 ). c. Control( ). Server Status( ) WLS1. Start( ). Oracle Solaris netstat SDP SPARC SuperCluster T4-4 Oracle Exalogic Elastic Cloud Software Oracle Solaris 11 netstat SDP. Oracle Solaris 11 157

Exalogic SDP InfiniBand netstat Oracle Solaris 11 SDP. root. # netstat -f sdp -s l, SDP ( ). SDP sdpactiveopens = 66357 sdpcurrestab = 748 sdpprfails = 0 sdprejects = 0 sdpoutsegs =39985638793 sdpindatabytes =9450383834191 sdpoutdatabytes =6228930927986 SDP sdpactiveopens = 0 sdpcurrestab = 0 sdpprfails = 0 sdprejects = 0 sdpinsegs = 14547 sdpoutsegs = 14525 sdpindatabytes =3537194 sdpoutdatabytes =2470907 Exalogic SDP InfiniBand SPARC SuperCluster T4-4 Oracle Exalogic Elastic Cloud Software. InfiniBand SDP. SDP SPARC SuperCluster T4-4. 158 InfiniBand SDP InfiniBand SDP Oracle RAC 11g 2,. Infiniband Oracle Exalogic Elastic Cloud Software IP Infiniband. 2. SPARC SuperCluster T4-4 2. 1 /etc/hosts InfiniBand IP. IP.. # Added for Listener over IB 158 SPARC SuperClusterT4-4 2012 12

Exalogic SDP InfiniBand 192.168.10.21 ssc01db01-ibvip.mycompany.com ssc01db01-ibvip 192.168.10.22 ssc01db02-ibvip.mycompany.com ssc01db02-ibvip 2 3 4 5 6 root InfiniBand. # /u01/app/grid/product/11.2.0.2/bin/srvctl add network -k 2 -S 192.168.10.0/255.255.255.0/bondib0. # /u01/app/grid/product/11.2.0.2/bin/crsctl stat res -t grep net ora.net1.network ora.net2.network -- Output indicating new Network resource # /u01/app/grid/product/11.2.0.2/bin/srvctl config network -k 2 Network exists: 2/192.168.10.0/255.255.255.0/bondib0, type static -- Output indicating Network resource on the 192.168.10.0 subnet 2 IP. srvctl add vip -n ssc01db01 -A ssc01db01-ibvip/255.255.255.0/bondib0 -k 2 srvctl add vip -n ssc01db02 -A ssc01db02-ibvip/255.255.255.0/bondib0 -k 2 "oracle" (Grid Infrastructure ), 3 VIP. srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522,/SDP:1522 ( InfiniBand) listener_networks init. tnsnames, $ORACLE_HOME/network/admin tnsnames.ora. TNSNAMES.ORA GRID_HOME. tnsnames.ora. IP. LISTENER_IBREMOTE. DBM_IB. 159

Exalogic SDP InfiniBand TNSNAMES. init.ora (LISTENER_NETWORKS) ALTER SYSTEM SET LISTENER_NETWORKS. (DESCRIPTION = DBM = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01-scan)(port = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) )) DBM_IB = (DESCRIPTION = (LOAD_BALANCE=on) (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01db01-ibvip)(port = 1522)) (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01db02-ibvip)(port = 1522)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = dbm) )) LISTENER_IBREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01db02-ibvip.mycompany.com)(port = 1522)) )) LISTENER_IBLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01db01-ibvip.mycompany.com)(port = 1522)) (ADDRESS = (PROTOCOL = SDP)(HOST = ssc01db01-ibvip.mycompany.com)(port = 1522)) )) LISTENER_IPLOCAL = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc0101-vip.mycompany.com)(port = 1521)) )) LISTENER_IPREMOTE = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = ssc01-scan.mycompany.com)(port = 1521)) )) 160 SPARC SuperClusterT4-4 2012 12

Exalogic SDP InfiniBand 7 8 listener_networks init. sysdba. SQLPLUS> alter system set listener_networks= ((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE)), ((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE)) scope=both; 7 LISTENER_IB. srvctl stop listener -l LISTENER_IB srvctl start listener -l LISTENER_IB 161

162

SPARC SuperCluster T4-4. 163 169 InfiniBand 175 180 Sun ZFS Storage 7320 180 PDU 181 3 PDU SPARC SuperCluster T4-4 IIB(InfiniBand) IB.IB 169 InfiniBand.. 1Gb/s Cat 5E 175.. Sun Datacenter InfiniBand Switch 36-12 IB ( U1) IB 1( U18) IB 2( U24) SPARC T4-4 - 13 SPARC T4-4 1( 8) SPARC T4-4 2( 13) SPARC T4-4 3( 27)( ) SPARC T4-4 4( 32)( ) Sun ZFS Storage 7320-14 Sun ZFS Storage 7320 1( 25) Sun ZFS Storage 7320 2( 26) Exadata Storage Server - 15 Exadata Storage Server 1( 2) 163

Exadata Storage Server 2( 4) Exadata Storage Server 3( 6) Exadata Storage Server 4( 37)( ) Exadata Storage Server 5( 39)( ) Exadata Storage Server 6( 41)( ) - 16 PDU AC - 17 12 Sun Datacenter InfiniBand Switch 36 1:NETMGT0 NET MGT 1 2: InfiniBand 0A-17A( ) 3: InfiniBand 0B-17B( ) 164 SPARC SuperClusterT4-4 2012 12

참조 도면 그림 13 SPARC T4-4 서버 그림 범례 1: NET MGT, 네트워크 관리 포트 2: SER MGT, 직렬 관리 포트 3: SER MGT 포트부터 터미널 장치까지의 RJ-45 케이블 - 어댑터가 필요할 수 있습니다. 4: RJ-45 케이블 - SPARC T4-4 서비스 프로세서를 이더넷에 연결합니다. 5: RJ-45 케이블 - 네트워크 포트 NET 0-3을 이더넷에 연결합니다. 6: 전원 코드 165

14 Sun ZFS Storage 7320 1: 1 2: 2 3: LED 4: 5: 6: NET0,1,2,3 7: USB 0, 1 8: HD15 15 Exadata Storage Server 1: 1 2: 2 3: LED 4: 5: 6: NET0,1,2,3 7: USB 0, 1 8: HD15 166 SPARC SuperClusterT4-4 2012 12

16 (Cisco Catalyst 4948 ) 1: 2: 1-16, 10/100/1000BASE-T 3: 17-32, 10/100/1000BASE-T 4: 33-48, 10/100/1000BASE-T 5:CON( ), MGT( ) 6: 45-48, 10 167

참조 도면 그림 17 PDU 회로 차단기 및 AC 소켓 그림 범례 1: PDU A 2: PDU B 168 SPARC SuperCluster T4-4 2012년 12월

InfiniBand InfiniBand IB(InfiniBand). IB - 14. IB 1-15. - 16. IB 2-17. - 18. 1Gb/s 175. IB 14 IB (U1) U1 1B 1 U18 8B U1 0B 2 U24 8B IB 1 15 -IB 1(U18) 1 U18 0A T4-4 4 U32 PCIE-12 P2 U18 0B T4-4 4 U32 PCIE-9 P2 U18 1A T4-4 4 U32 PCIE-7 P2 U18 1B T4-4 4 U32 PCIE-1 P2 U18 2A U18 2B U18 3A Sun ZFS Storage 7320 2 U26 PCIE-0 P1 U18 3B T4-4 2 U13 PCIE-12 P1 U18 4A T4-4 2 U13 PCIE-9 P1 169

InfiniBand 15 -IB 1(U18) ( ) 1 U18 4B T4-4 2 U13 PCIE-7 P1 U18 5A T4-4 2 U13 PCIE-1 P1 U18 5B T4-4 3 U27 PCIE-12 P2 U18 6A T4-4 3 U27 PCIE-9 P2 U18 6B T4-4 3 U27 PCIE-7 P2 U18 7A T4-4 3 U27 PCIE-1 P2 U18 7B U18 8A IB 2 U24 8A U18 8B IB U1 1B U18 9A IB 2 U24 9B U18 9B IB 2 U24 9A U18 10A IB 2 U24 10B U18 10B IB 2 U24 10A U18 11A IB 2 U24 11B U18 11B IB 2 U24 11A U18 12A U18 12B T4-4 1 U8 PCIE-12 P1 U18 13A T4-4 1 U8 PCIE-9 P1 U18 13B T4-4 1 U8 PCIE-7 P1 U18 14A T4-4 1 U8 PCIE-1 P1 U18 14B Sun ZFS Storage 7320 1 U25 PCIE-0 P1 U18 15A Exadata Storage Server 6 U41 PCIE-3 P2 U18 15B Exadata Storage Server 5 U39 PCIE-3 P2 U18 16A Exadata Storage Server 4 U37 PCIE-3 P2 U18 16B Exadata Storage Server 3 U6 PCIE-3 P1 U18 17A Exadata Storage Server 2 U4 PCIE-3 P1 U18 17B Exadata Storage Server 1 U2 PCIE-3 P1 170 SPARC SuperClusterT4-4 2012 12

InfiniBand 16 -IB 1(U18) 1 U18 0A U18 0B U18 1A U18 1B U18 2A U18 2B U18 3A Sun ZFS Storage 7320 2 U26 PCIE-0 P1 U18 3B T4-4 2 U13 PCIE-12 P1 U18 4A T4-4 2 U13 PCIE-9 P1 U18 4B T4-4 2 U13 PCIE-7 P1 U18 5A T4-4 2 U13 PCIE-1 P1 U18 5B U18 6A U18 6B U18 7A U18 7B U18 8A IB 2 U24 8A U18 8B IB U1 1B U18 9A IB 2 U24 9B U18 9B IB 2 U24 9A U18 10A IB 2 U24 10B U18 10B IB 2 U24 10A U18 11A IB 2 U24 11B U18 11B IB 2 U24 11A U18 12A U18 12B T4-4 1 U8 PCIE-12 P1 171

InfiniBand 16 -IB 1(U18) ( ) 1 U18 13A T4-4 1 U8 PCIE-9 P1 U18 13B T4-4 1 U8 PCIE-7 P1 U18 14A T4-4 1 U8 PCIE-1 P1 U18 14B Sun ZFS Storage 7320 1 U25 PCIE-0 P1 U18 15A U18 15B U18 16A U18 16B Exadata Storage Server 3 U6 PCIE-3 P1 U18 17A Exadata Storage Server 2 U4 PCIE-3 P1 U18 17B Exadata Storage Server 1 U2 PCIE-3 P1 IB 2 17 -IB 2(U24) 2 U24 0A T4-4 4 U32 PCIE-12 P1 U24 0B T4-4 4 U32 PCIE-9 P1 U24 1A T4-4 4 U32 PCIE-7 P1 U24 1B T4-4 4 U32 PCIE-1 P1 U24 2A U24 2B U24 3A Sun ZFS Storage 7320 2 U26 PCIE-0 P2 U24 3B T4-4 2 U13 PCIE-12 P2 U24 4A T4-4 2 U13 PCIE-9 P2 U24 4B T4-4 2 U13 PCIE-7 P2 U24 5A T4-4 2 U13 PCIE-1 P2 U24 5B T4-4 3 U27 PCIE-12 P1 172 SPARC SuperClusterT4-4 2012 12

InfiniBand 17 -IB 2(U24) ( ) 2 U24 6A T4-4 3 U27 PCIE-9 P1 U24 6B T4-4 3 U27 PCIE-7 P1 U24 7A T4-4 3 U27 PCIE-1 P1 U24 7B U24 8A IB 1 U18 8A U24 8B IB U1 0B U24 9A IB 1 U18 9B U24 9B IB 1 U18 9A U24 10A IB 1 U18 10B U24 10B IB 1 U18 10A U24 11A IB 1 U18 11B U24 11B IB 1 U18 11A U24 12A U24 12B T4-4 1 U8 PCIE-12 P2 U24 13A T4-4 1 U8 PCIE-9 P2 U24 13B T4-4 1 U8 PCIE-7 P2 U24 14A T4-4 1 U8 PCIE-1 P2 U24 14B Sun ZFS Storage 7320 1 U25 PCIE-0 P2 U24 15A Exadata Storage Server 6 U41 PCIE 3 P1 U24 15B Exadata Storage Server 5 U39 PCIE 3 P1 U24 16A Exadata Storage Server 4 U37 PCIE 3 P1 U24 16B Exadata Storage Server 3 U6 PCIE-3 P2 U24 17A Exadata Storage Server 2 U4 PCIE-3 P2 U24 17B Exadata Storage Server 1 U2 PCIE-3 P2 173

InfiniBand 18 -IB 2(U24) 2 U24 0A U24 0B U24 1A U24 1B U24 2A U24 2B U24 3A Sun ZFS Storage 7320 2 U26 PCIE-0 P2 U24 3B T4-4 2 U13 PCIE-12 P2 U24 4A T4-4 2 U13 PCIE-9 P2 U24 4B T4-4 2 U13 PCIE-7 P2 U24 5A T4-4 2 U13 PCIE-1 P2 U24 5B U24 6A U24 6B U24 7A U24 7B U24 8A IB 1 U18 8A U24 8B IB U1 0B U24 9A IB 1 U18 9B U24 9B IB 1 U18 9A U24 10A IB 1 U18 10B U24 10B IB 1 U18 10A U24 11A IB 1 U18 11B U24 11B IB 1 U18 11A U24 12A U24 12B T4-4 1 U8 PCIE-12 P2 174 SPARC SuperClusterT4-4 2012 12

18 -IB 2(U24) ( ) 2 U24 13A T4-4 1 U8 PCIE-9 P2 U24 13B T4-4 1 U8 PCIE-7 P2 U24 14A T4-4 1 U8 PCIE-1 P2 U24 14B Sun ZFS Storage 7320 1 U25 PCIE-0 P2 U24 15A U24 15B U24 16A U24 16B Exadata Storage Server 3 U6 PCIE-3 P2 U24 17A Exadata Storage Server 2 U4 PCIE-3 P2 U24 17B Exadata Storage Server 1 U2 PCIE-3 P2 (Cisco Catalyst 4948 10, 16) U19 SPARC SuperCluster T4-4. SPARC T4-4, Sun ZFS Storage 7320, Exadata Storage Server PDU. 19. 20. 45, 46, 47 48 4(10BASE-T/100BASE-TX/1000BASE-T ) 6(1000BASE-X ). 1, 2, 3 4. 175

19 Cat 5E - (U19) 1 2 3 4 5 T4-4 4 U32 NET 3,3m 6 T4-4 3 U27 NET 3,3m 7 T4-4 4 U32 NET 2,3m 8 T4-4 3 U27 NET 2,3m 9 T4-4 4 U32 NET 1,3m 10 T4-4 3 U27 NET 1,3m 11 T4-4 4 U32 NET 0,3m 12 T4-4 4 U32 NET MGT,3m 13 T4-4 3 U27 NET-0,3m 14 T4-4 3 U27 NET MGT,3m 15 PDU-A PDU-A NET MGT,1m 16 T4-4 2 U13 NET-1,3m 17 T4-4 2 U13 NET-0,3m 18 T4-4 2 U13 NET MGT,3m 19 PDU-B PDU-B NET MGT,1m 20 T4-4 2 U13 NET-2,3m 21 T4-4 1 U8 NET-0,3m 22 T4-4 1 U8 NETMGT,3m 23 Sun ZFS Storage 7320 2 U26 NET-1,3m 24 T4-4 2 U13 NET-3,3m 25 Sun ZFS Storage 7320 2 U26 NET-0,3m 176 SPARC SuperClusterT4-4 2012 12

19 Cat 5E - (U19) ( ) 26 T4-4 1 U8 NET-1,3m 27 Sun ZFS Storage 7320 2 U26 NET-2,3m 28 T4-4 1 U8 NET-2,3m 29 Sun ZFS Storage 7320 1 U25 NET-1,3m 30 T4-4 1 U8 NET-3,3m 31 Sun ZFS Storage 7320 1 U25 NET-0,3m 32 Sun ZFS Storage 7320 1 U25 NET-2,3m 33 Exadata Storage Server 6 U41 NET-0,3m 34 Exadata Storage Server 6 U41 NET MGT,3m 35 Exadata Storage Server 5 U39 NET-0,3m 36 Exadata Storage Server 5 U39 NET MGT,3m 37 Exadata Storage Server 4 U37 NET-0,3m 38 Exadata Storage Server 4 U37 NET MGT,3m 39 Exadata Storage Server 3 U6 NET-0,3m 40 Exadata Storage Server 3 U6 NETMGT,3m 41 Exadata Storage Server 2 U4 NET-0,3m 42 Exadata Storage Server 2 U4 NETMGT,3m 43 Exadata Storage Server 1 U2 NET-0,3m 44 Exadata Storage Server 1 U2 NETMGT,3m 45 Sun Datacenter InfiniBand Switch 36 2 U24 NET-0,3m 46 Sun Datacenter InfiniBand Switch 36 1 U18 NET-0,3m 47 Sun Datacenter Infiniband Switch 36 U1 NET-0,3m 48 ( ),3m 177

20 Cat 5E - (U19) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 PDU-A PDU-A NET MGT,1m 16 T4-4 2 U13 NET-1,3m 17 T4-4 2 U13 NET-0,3m 18 T4-4 2 U13 NET MGT,3m 19 PDU-B PDU-B NET MGT,1m 20 T4-4 2 U13 NET-2,3m 21 T4-4 1 U8 NET-0,3m 22 T4-4 1 U8 NETMGT,3m 23 Sun ZFS Storage 7320 2 U26 NET-1,3m 24 T4-4 2 U13 NET-3,3m 25 Sun ZFS Storage 7320 2 U26 NET-0,3m 178 SPARC SuperClusterT4-4 2012 12

20 Cat 5E - (U19) ( ) 26 T4-4 1 U8 NET-1,3m 27 Sun ZFS Storage 7320 2 U26 NET-2,3m 28 T4-4 1 U8 NET-2,3m 29 Sun ZFS Storage 7320 1 U25 NET-1,3m 30 T4-4 1 U8 NET-3,3m 31 Sun ZFS Storage 7320 1 U25 NET-0,3m 32 Sun ZFS Storage 7320 1 U25 NET-2,3m 33 34 35 36 37 38 39 Exadata Storage Server 3 U6 NET-0,3m 40 Exadata Storage Server 3 U6 NETMGT,3m 41 Exadata Storage Server 2 U4 NET-0,3m 42 Exadata Storage Server 2 U4 NETMGT,3m 43 Exadata Storage Server 1 U2 NET-0,3m 44 Exadata Storage Server 1 U2 NETMGT,3m 45 Sun Datacenter InfiniBand Switch 36 2 U24 NET-0,3m 46 Sun Datacenter InfiniBand Switch 36 1 U18 NET-0,3m 47 Sun Datacenter Infiniband Switch 36 U1 NET-0,3m 48 ( ),3m 179

Sun ZFS Storage 7320 Sun ZFS Storage 7320 21 Sun ZFS Storage 7320.. Sun ZFS Storage 7320 1( 25) Sun ZFS Storage 7320 2( 26) Sun Disk Shelf( 20) 21 Sun ZFS Storage 7320 SAS U25 PCIE-1 P0 U26 PCIE-1 P2,7 U25 PCIE-1 P1 U26 PCIE-1 P1,7 U25 PCIE-1 P2 U26 PCIE-1 P0,7 U26 PCIE-2 P1 U20 SAS 1,2m U25 PCIE-2 P1 U20 SAS 0,2m U26 PCIE-2 P0 U20 SAS 1,2m U25 PCIE-2 P0 U20 SAS 0,2m PDU AC 22 (Power Distribution Unit, PDU). 2. PDU PDU.. PDU PDU AC 17. 22 PDU PDU-A/PSU-0 - PDB-B/PDU-1 - PDU-A/PSU-2 - PDB-B/PSU-3 - U41 5-4 0-2 Exadata Storage Server 6 U39 5-3 0-3 Exadata Storage Server 5 180 SPARC SuperClusterT4-4 2012 12

3 PDU 22 PDU ( ) PDB-B/PDU-1 PDU-A/PSU-0 - - PDU-A/PSU-2 - PDB-B/PSU-3 - U37 5-2 0-4 Exadata Storage Server 4 U32 4-2 1-4 5-0 0-6 SPARC T4-4 4 U27 3-6 2-0 4-1 1-5 SPARC T4-4 3 U26 3-4 2-2 Sun ZFS Storage 7320 2 U25 3-2 2-4 Sun ZFS Storage 7320 1 U24 3-0 2-6 Sun Datacenter InfiniBand Switch 36 2 U20 2-6 3-0 Sun Disk Shelf U19 2-5 3-1 U18 2-3 3-3 Sun Datacenter InfiniBand Switch 36 1 U13 1-1 4-5 2-1 3-5 SPARC T4-4 2 U8 0-6 5-0 1-0 4-6 SPARC T4-4 1 U6 0-4 5-2 Exadata Storage Server 3 U4 0-3 5-3 Exadata Storage Server 2 U2 0-2 5-4 Exadata Storage Server 1 U1 0-1 5-5 Sun Datacenter Infiniband Switch 36 3 PDU 3 AC 23 (Power Distribution Unit, PDU). 2. PDU PDU.. PDU PDU AC 17. 181

3 PDU 23 3 PDU PDU-A/PSU-0 - PDB-B/PDU-1 - PDU-A/PSU-2 - PDB-B/PSU-3 - U41 5-6 2-0 Exadata Storage Server 6 U39 5-4 2-2 Exadata Storage Server 5 U37 5-2 2-4 Exadata Storage Server 4 U32 4-2 1-4 5-0 2-6 SPARC T4-4 4 U27 3-6 0-0 4-1 1-5 SPARC T4-4 3 U26 3-4 0-2 Sun ZFS Storage 7320 2 U25 3-2 0-4 Sun ZFS Storage 7320 1 U24 3-0 0-6 Sun Datacenter InfiniBand Switch 36 2 U20 2-6 5-0 Sun Disk Shelf U19 2-5 5-1 U18 2-4 5-2 Sun Datacenter InfiniBand Switch 36 1 U13 1-1 4-5 2-0 5-6 SPARC T4-4 2 U8 0-6 3-0 1-0 4-6 SPARC T4-4 1 U6 0-4 3-2 Exadata Storage Server 3 U4 0-3 3-3 Exadata Storage Server 2 U2 0-2 3-4 Exadata Storage Server 1 U1 0-1 3-5 Sun Datacenter Infiniband Switch 36 182 SPARC SuperClusterT4-4 2012 12

SPARC SuperClusterT4-4 SPARC SuperCluster T4-4 SPARC SuperCluster T4-4. 183 185 187 189 192 196 200 206 Sun Datacenter InfiniBand Switch 36. SPARC SuperCluster T4-4 Sun Datacenter InfiniBand Switch 36. InfiniBand QSFP(Quad Small Form-factor Pluggable)... 1(U1). 18 (U18) 24(U24). 7... 183

7. 8.,. 184 SPARC SuperClusterT4-4 2012 12

, 1. 2 1. 8 2 8 SPARC SuperCluster T4-4. U1 U18 U24 2 (R1-U1). 185

24 1 R1-U24 R1-U24 2 1 R1-U18 R1-U18 2 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R1-U1-P5A R1-U24-P9B ~ R1-U1-P6A R1-U24-P10A ~ R2-U1-P7A R1-U24-P10B ~ R2-U1-P8A R1-U24-P11A ~ R2-U1-P9A R1-U24-P11B ~ R2-U1-P10A R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R1-U1-P5B R1-U18-P9B ~ R1-U1-P6B R1-U18-P10A ~ R2-U1-P7B R1-U18-P10B ~ R2-U1-P8B R1-U18-P11A ~ R2-U1-P9B R1-U18-P11B ~ R2-U1-P10B (R2-U1). 25 2 R2-U24 R2-U24 1 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A R2-U24-P9A ~ R2-U1-P5A R2-U24-P9B ~ R2-U1-P6A R2-U24-P10A ~ R1-U1-P7A R2-U24-P10B ~ R1-U1-P8A R2-U24-P11A ~ R1-U1-P9A R2-U24-P11B ~ R1-U1-P10A 186 SPARC SuperClusterT4-4 2012 12

25 ( ) 2 R2-U18 R2-U18 1 R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B R2-U18-P9A ~ R2-U1-P5B R2-U18-P9B ~ R2-U1-P6B R2-U18-P10A ~ R1-U1-P7B R2-U18-P10B ~ R1-U1-P8B R2-U18-P11A ~ R1-U1-P9B R2-U18-P11B ~ R1-U1-P10B (R1-U1). 26 1 R1-U24 R1-U24 2 R1-U24 3 1 R1-U18 R1-U18 2 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R1-U1-P5A R1-U24-P9B ~ R2-U1-P6A R1-U24-P10A ~ R2-U1-P7A R1-U24-P10B ~ R2-U1-P8A R1-U24-P11A ~ R3-U1-P9A R1-U24-P11B ~ R3-U1-P10A R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R1-U1-P5B R1-U18-P9B ~ R2-U1-P6B R1-U18-P10A ~ R2-U1-P7B R1-U18-P10B ~ R2-U1-P8B 187

26 ( ) R1-U18 3 R1-U18-P11A ~ R3-U1-P9B R1-U18-P11B ~ R3-U1-P10B (R2-U1). 27 2 R2-U24 R2-U24 1 R2-U24 3 2 R2-U18 R2-U18 1 R2-U18 3 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A R2-U24-P9A ~ R2-U1-P5A R2-U24-P11A ~ R1-U1-P9A R2-U24-P11B ~ R1-U1-P10A R2-U24-P9B ~ R3-U1-P6A R2-U24-P10A ~ R3-U1-P7A R2-U24-P10B ~ R3-U1-P8A R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B R2-U18-P9A ~ R2-U1-P5B R2-U18-P11A ~ R1-U1-P9B R2-U18-P11B ~ R1-U1-P10B R2-U18-P9B ~ R3-U1-P6B R2-U18-P10A ~ R3-U1-P7B R2-U18-P10B ~ R3-U1-P8B (R3-U1). 188 SPARC SuperClusterT4-4 2012 12

28 3 R3-U24 R3-U24 1 R3-U24 2 3 R3-U18 R3-U18 1 R3-U18 2 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A R3-U24-P9A ~ R3-U1-P5A R3-U24-P9B ~ R1-U1-P6A R3-U24-P10A ~ R1-U1-P7A R3-U24-P10B ~ R1-U1-P8A R3-U24-P11A ~ R2-U1-P9A R3-U24-P11B ~ R2-U1-P10A R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B R3-U18-P9A ~ R3-U1-P5B R3-U18-P9B ~ R1-U1-P6B R3-U18-P10A ~ R1-U1-P7B R3-U18-P10B ~ R1-U1-P8B R3-U18-P11A ~ R2-U1-P9B R3-U18-P11B ~ R2-U1-P10B (R1-U1). 29 1 R1-U24 R1-U24 2 R1-U24 3 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R2-U1-P5A R1-U24-P9B ~ R2-U1-P6A R1-U24-P10A ~ R3-U1-P7A R1-U24-P10B ~ R3-U1-P8A 189

29 ( ) R1-U24 4 1 R1-U18 R1-U18 2 R1-U18 3 R1-U18 4 R1-U24-P11A ~ R4-U1-P9A R1-U24-P11B ~ R4-U1-P10A R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R2-U1-P5B R1-U18-P9B ~ R2-U1-P6B R1-U18-P10A ~ R3-U1-P7B R1-U18-P10B ~ R3-U1-P8B R1-U18-P11A ~ R4-U1-P9B R1-U18-P11B ~ R4-U1-P10B 10 10 (R2-U1). 30 2 R2-U24 R2-U24 1 R2-U24 3 R2-U24 4 2 R2-U18 R2-U18 1 R2-U18 3 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A R2-U24-P11A ~ R1-U1-P9A R2-U24-P11B ~ R1-U1-P10A R2-U24-P9A ~ R3-U1-P5A R2-U24-P9B ~ R3-U1-P6A R2-U24-P10A ~ R4-U1-P7A R2-U24-P10B ~ R4-U1-P8A R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B R2-U18-P11A ~ R1-U1-P9B R2-U18-P11B ~ R1-U1-P10B R2-U18-P9A ~ R3-U1-P5B R2-U18-P9B ~ R3-U1-P6B 190 SPARC SuperClusterT4-4 2012 12

30 ( ) R2-U18 4 R2-U18-P10A ~ R4-U1-P7B R2-U18-P10B ~ R4-U1-P8B (R3-U1). 31 3 R3-U24 R3-U24 1 R3-U24 2 R3-U24 4 3 R3-U18 R3-U18 1 R3-U18 2 R3-U18 4 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A R3-U24-P10A ~ R1-U1-P7A R3-U24-P10B ~ R1-U1-P8A R3-U24-P11A ~ R2-U1-P9A R3-U24-P11B ~ R2-U1-P10A R3-U24-P9A ~ R4-U1-P5A R3-U24-P9B ~ R4-U1-P6A R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B R3-U18-P10A ~ R1-U1-P7B R3-U18-P10B ~ R1-U1-P8B R3-U18-P11A ~ R2-U1-P9B R3-U18-P11B ~ R2-U1-P10B R3-U18-P9A ~ R4-U1-P5B R3-U18-P9B ~ R4-U1-P6B (R4-U1). 191

32 4 R4-U24 R4-U24 1 R4-U24 2 R4-U24 3 4 R4-U18 R4-U18 1 R4-U18 2 R4-U18 3 R4-U24-P8A ~ R4-U1-P3A R4-U24-P8B ~ R4-U1-P4A R4-U24-P9A ~ R1-U1-P5A R4-U24-P9B ~ R1-U1-P6A R4-U24-P10A ~ R2-U1-P7A R4-U24-P10B ~ R2-U1-P8A R4-U24-P11A ~ R3-U1-P9A R4-U24-P11B ~ R3-U1-P10A R4-U18-P8A ~ R4-U1-P3B R4-U18-P8B ~ R4-U1-P4B R4-U18-P9A ~ R1-U1-P5B R4-U18-P9B ~ R1-U1-P6B R4-U18-P10A ~ R2-U1-P7B R4-U18-P10B ~ R2-U1-P8B R4-U18-P11A ~ R3-U1-P9B R4-U18-P11B ~ R3-U1-P10B 10 10 (R1-U1). 33 1 R1 U24 R1 U24 2 R1 U24 3 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R2-U1-P5A R1-U24-P9B ~ R2-U1-P6A R1-U24-P10A ~ R3-U1-P7A R1-U24-P10B ~ R3-U1-P8A 3 192 SPARC SuperClusterT4-4 2012 12

33 ( ) R1 U24 4 R1-U24-P11A ~ R4-U1-P9A 10 R1 U24 5 R1-U24-P11B ~ R5-U1-P10A 10 1 R1 U18 R1 U18 2 R1 U18 3 R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R2-U1-P5B R1-U18-P9B ~ R2-U1-P6B R1-U18-P10A ~ R3-U1-P7B R1-U18-P10B ~ R3-U1-P8B 3 3 R1 U18 4 R1-U18-P11A ~ R4-U1-P9B 10 R1 U18 5 R1-U18-P11B ~ R5-U1-P10B 10 (R2-U1). 34 2 R2 U24 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A 3 R2 U24 1 R2-U24-P11B ~ R1-U1-P10A R2 U24 3 R2 U24 4 R2-U24-P9A ~ R3-U1-P5A R2-U24-P9B ~ R3-U1-P6A R2-U24-P10A ~ R4-U1-P7A R2-U24-P10B ~ R4-U1-P8A R2 U24 5 R2-U24-P11A ~ R5-U1-P9A 10 2 R2 U18 R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B 3 R2 U18 1 R2-U18-P11B ~ R1-U1-P10B R2 U18 3 R2-U18-P9A ~ R3-U1-P5B R2-U18-P9B ~ R3-U1-P6B 193

34 ( ) R2 U18 4 R2-U18-P10A ~ R4-U1-P7B R2-U18-P10B ~ R4-U1-P8B R2 U18 5 R2-U18-P11A ~ R5-U1-P9B 10 (R3-U1). 35 3 R3 U24 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A 3 R3 U24 1 R3-U24-P11A ~ R1-U1-P9A R3 U24 2 R3-U24-P11B ~ R2-U1-P10A R3 U24 4 R3 U24 5 3 R3 U18 R3-U24-P9A ~ R4-U1-P5A R3-U24-P9B ~ R4-U1-P6A R3-U24-P10A ~ R5-U1-P7A R3-U24-P10B ~ R5-U1-P8A R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B 3 R3 U18 1 R3-U18-P11A ~ R1-U1-P9B R3 U18 2 R3-U18-P11B ~ R2-U1-P10B R3 U18 4 R3 U18 5 R3-U18-P9A ~ R4-U1-P5B R3-U18-P9B ~ R4-U1-P6B R3-U18-P10A ~ R5-U1-P7B R3-U18-P10B ~ R5-U1-P8B (R4-U1). 194 SPARC SuperClusterT4-4 2012 12

36 4 R4 U24 R4 U24 1 R4-U24-P8A ~ R4-U1-P3A R4-U24-P8B ~ R4-U1-P4A R4-U24-P10A ~ R1-U1-P7A R4-U24-P10B ~ R1-U1-P8A 3 10 R4 U24 2 R4-U24-P11A ~ R2-U1-P9A R4 U24 3 R4-U24-P11B ~ R3-U1-P10A R4 U24 5 4 R4 U18 R4 U18 1 R4-U24-P9A ~ R5-U1-P5A R4-U24-P9B ~ R5-U1-P6A R4-U18-P8A ~ R4-U1-P3B R4-U18-P8B ~ R4-U1-P4B R4-U18-P10A ~ R1-U1-P7B R4-U18-P10B ~ R1-U1-P8B 3 10 R4 U18 2 R4-U18-P11A ~ R2-U1-P9B R4 U18 3 R4-U18-P11B ~ R3-U1-P10B R4 U18 5 R4-U18-P9A ~ R5-U1-P5B R4-U18-P9B ~ R5-U1-P6B (R5-U1). 37 5 R5 U24 R5 U24 1 R5 U24 2 R5-U24-P8A ~ R5-U1-P3A R5-U24-P8B ~ R5-U1-P4A R5-U24-P9A ~ R1-U1-P5A R5-U24-P9B ~ R1-U1-P6A R5-U24-P10A ~ R2-U1-P7A R5-U24-P10B ~ R2-U1-P8A 3 10 10 R5 U24 3 R5-U24-P11A ~ R3-U1-P9A 195

37 ( ) R5 U24 4 R5-U24-P11B ~ R4-U1-P10A 5 R5 U18 R5 U18 1 R5 U18 2 R5-U18-P8A ~ R5-U1-P3B R5-U18-P8B ~ R5-U1-P4B R5-U18-P9A ~ R1-U1-P5B R5-U18-P9B ~ R1-U1-P6B R5-U18-P10A ~ R2-U1-P7B R5-U24-P10B ~ R2-U1-P8B 3 10 10 R5 U18 3 R5-U18-P11A ~ R3-U1-P9B R5 U18 4 R5-U18-P11B ~ R4-U1-P10B (R1-U1). 38 1 R1 U24 R1 U24 2 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A R1-U24-P9A ~ R2-U1-P5A R1-U24-P9B ~ R2-U1-P6A 3 R1 U24 3 R1-U24-P10A ~ R3-U1-P7A R1 U24 4 R1-U24-P10B ~ R4-U1-P8A 10 R1 U24 5 R1-U24-P11A ~ R5-U1-P9A 10 R1 U24 6 R1-U24-P11B ~ R6-U1-P10A 10 1 R1 U18 R1 U18 2 R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B R1-U18-P9A ~ R2-U1-P5B R1-U18-P9B ~ R2-U1-P6B 3 R1 U18 3 R1-U18-P10A ~ R3-U1-P7B 196 SPARC SuperClusterT4-4 2012 12

38 ( ) R1 U18 4 R1-U18-P10B ~ R4-U1-P8B 10 R1 U18 5 R1-U18-P11A ~ R5-U1-P9B 10 R1 U18 6 R1-U18-P11B ~ R6-U1-P10B 10 (R2-U1). 39 2 R2 U24 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A 3 R2 U24 1 R2-U24-P11B ~ R1-U1-P10A R2 U24 3 R2-U24-P9A ~ R3-U1-P5A R2-U24-P9B ~ R3-U1-P6A R2 U24 4 R2-U24-P10A ~ R4-U1-P7A R2 U24 5 R2-U24-P10B ~ R5-U1-P8A 10 R2 U24 6 R2-U24-P11A ~ R6-U1-P9A 10 2 R2 U18 R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B 3 R2 U18 1 R2-U18-P11B ~ R1-U1-P10B R2 U18 3 R2-U18-P9A ~ R3-U1-P5B R2-U18-P9B ~ R3-U1-P6B R2 U18 4 R2-U18-P10A ~ R4-U1-P7B R2 U18 5 R2-U18-P10B ~ R5-U1-P8B 10 R2 U18 6 R2-U18-P11A ~ R6-U1-P9B 10 (R3-U1). 197

40 3 R3 U24 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A 3 R3 U24 1 R3-U24-P11A ~ R1-U1-P9A R3 U24 2 R3-U24-P11B ~ R2-U1-P10A R3 U24 4 R3-U24-P9A ~ R4-U1-P5A R3-U24-P9B ~ R4-U1-P6A R3 U24 5 R3-U24-P10A ~ R5-U1-P7A R3 U24 6 R3-U24-P10B ~ R6-U1-P8A 3 R3 U18 R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B 3 R3 U18 1 R3-U18-P11A ~ R1-U1-P9B R3 U18 2 R3-U18-P11B ~ R2-U1-P10B R3 U18 4 R3-U18-P9A ~ R4-U1-P5B R3-U18-P9B ~ R4-U1-P6B R3 U18 5 R3-U18-P10A ~ R5-U1-P7B R3 U18 6 R3-U18-P10B ~ R6-U1-P8B (R4-U1). 41 4 R4 U24 R4-U24-P8A ~ R4-U1-P3A R4-U24-P8B ~ R4-U1-P4A 3 R4 U24 1 R4-U24-P10B ~ R1-U1-P8A 10 R4 U24 2 R4-U24-P11A ~ R2-U1-P9A R4 U24 3 R4-U24-P11B ~ R3-U1-P10A R4 U24 5 R4-U24-P9A ~ R5-U1-P5A R4-U24-P9B ~ R5-U1-P6A 198 SPARC SuperClusterT4-4 2012 12

41 ( ) R4 U24 6 R4-U24-P10A ~ R6-U1-P7A 4 R4 U18 R4-U18-P8A ~ R4-U1-P3B R4-U18-P8B ~ R4-U1-P4B 3 R4 U18 1 R4-U18-P10B ~ R1-U1-P8B 10 R4 U18 2 R4-U18-P11A ~ R2-U1-P9B R4 U18 3 R4-U18-P11B ~ R3-U1-P10B R4 U18 5 R4-U18-P9A ~ R5-U1-P5B R4-U18-P9B ~ R5-U1-P6B R4 U18 6 R4-U18-P10A ~ R6-U1-P7B (R5-U1). 42 5 R5 U24 R5-U24-P8A ~ R5-U1-P3A R5-U24-P8B ~ R5-U1-P4A 3 R5 U24 1 R5-U24-P10A ~ R1-U1-P7A 10 R5 U24 2 R5-U24-P10B ~ R2-U1-P8A 10 R5 U24 3 R5-U24-P11A ~ R3-U1-P9A R5 U24 4 R5-U24-P11B ~ R4-U1-P10A R5 U24 6 5 R5 U18 R5-U24-P9A ~ R6-U1-P5A R5-U24-P9B ~ R6-U1-P6A R5-U18-P8A ~ R5-U1-P3B R5-U18-P8B ~ R5-U1-P4B 3 R5 U18 1 R5-U18-P10A ~ R1-U1-P7B 10 R5 U18 2 R5-U24-P10B ~ R2-U1-P8B 10 R5 U18 3 R5-U18-P11A ~ R3-U1-P9B R5 U18 4 R5-U18-P11B ~ R4-U1-P10B 199

42 ( ) R5 U18 6 R5-U18-P9A ~ R6-U1-P5B R5-U18-P9B ~ R6-U1-P6B (R6-U1). 43 6 R6 U24 R6 U24 1 R6-U24-P8A ~ R6-U1-P3A R6-U24-P8B ~ R6-U1-P4A R6-U24-P9A ~ R1-U1-P5A R6-U24-P9B ~ R1-U1-P6A 3 10 R6 U24 2 R6-U24-P10A ~ R2-U1-P7A 10 R6 U24 3 R6-U24-P10B ~ R3-U1-P8A R6 U24 4 R6-U24-P11A ~ R4-U1-P9A R6 U24 5 R6-U24-P11B ~ R5-U1-P10A 6 R6 U18 R6-U18-P8A ~ R6-U1-P3B R6-U18-P8B ~ R6-U1-P4B 3 R6 U18 2 R6-U24-P10A ~ R2-U1-P7B 10 R6 U18 1 R6-U18-P9A ~ R1-U1-P5B R6-U18-P9B ~ R1-U1-P6B 10 R6 U18 3 R6-U18-P10B ~ R3-U1-P8B R6 U18 4 R6-U18-P11A ~ R4-U1-P9B R6 U18 5 R6-U18-P11B ~ R5-U1-P10B (R1-U1). 200 SPARC SuperClusterT4-4 2012 12

44 1 R1 U24 R1-U24-P8A ~ R1-U1-P3A R1-U24-P8B ~ R1-U1-P4A 3 R1 U24 2 R1-U24-P9A ~ R2-U1-P5A R1 U24 3 R1-U24-P9B ~ R3-U1-P6A R1 U24 4 R1-U24-P10A ~ R4-U1-P7A 10 R1 U24 5 R1-U24-P10B ~ R5-U1-P8A 10 R1 U24 6 R1-U24-P11A ~ R6-U1-P9A 10 R1 U24 7 R1-U24-P11B ~ R7-U1-P10A 10 1 R1 U18 R1-U18-P8A ~ R1-U1-P3B R1-U18-P8B ~ R1-U1-P4B 3 R1 U18 2 R1-U18-P9A ~ R2-U1-P5B R1 U18 3 R1-U18-P9B ~ R3-U1-P6B R1 U18 4 R1-U18-P10A ~ R4-U1-P7B 10 R1 U18 5 R1-U18-P10B ~ R5-U1-P8B 10 R1 U18 6 R1-U18-P11A ~ R6-U1-P9B 10 R1 U18 7 R1-U18-P11B ~ R7-U1-P10B 10 (R2-U1). 45 2 R2 U24 R2-U24-P8A ~ R2-U1-P3A R2-U24-P8B ~ R2-U1-P4A 3 R2 U24 1 R2-U24-P11B ~ R1-U1-P10A R2 U24 3 R2-U24-P9A ~ R3-U1-P5A R2 U24 4 R2-U24-P9B ~ R4-U1-P6A R2 U24 5 R2-U24-P10A ~ R5-U1-P7A 10 R2 U24 6 R2-U24-P10B ~ R6-U1-P8A 10 201

45 ( ) R2 U24 7 R2-U24-P11A ~ R7-U1-P9A 10 2 R2 U18 R2-U18-P8A ~ R2-U1-P3B R2-U18-P8B ~ R2-U1-P4B 3 R2 U18 1 R2-U18-P11B ~ R1-U1-P10B R2 U18 3 R2-U18-P9A ~ R3-U1-P5B R2 U18 4 R2-U18-P9B ~ R4-U1-P6B R2 U18 5 R2-U18-P10A ~ R5-U1-P7B 10 R2 U18 6 R2-U18-P10B ~ R6-U1-P8B 10 R2 U18 7 R2-U18-P11A ~ R7-U1-P9B 10 (R3-U1). 46 3 R3 U24 R3-U24-P8A ~ R3-U1-P3A R3-U24-P8B ~ R3-U1-P4A 3 R3 U24 1 R3-U24-P11A ~ R1-U1-P9A R3 U24 2 R3-U24-P11B ~ R2-U1-P10A R3 U24 4 R3-U24-P9A ~ R4-U1-P5A R3 U24 5 R3-U24-P9B ~ R5-U1-P6A R3 U24 6 R3-U24-P10A ~ R6-U1-P7A 10 R3 U24 7 R3-U24-P10B ~ R7-U1-P8A 10 3 R3 U18 R3-U18-P8A ~ R3-U1-P3B R3-U18-P8B ~ R3-U1-P4B 3 R3 U18 1 R3-U18-P11A ~ R1-U1-P9B R3 U18 2 R3-U18-P11B ~ R2-U1-P10B R3 U18 4 R3-U18-P9A ~ R4-U1-P5B R3 U18 5 R3-U18-P9B ~ R5-U1-P6B 202 SPARC SuperClusterT4-4 2012 12

46 ( ) R3 U18 6 R3-U18-P10A ~ R6-U1-P7B 10 R3 U18 7 R3-U18-P10B ~ R7-U1-P8B 10 (R4-U1). 47 4 R4 U24 R4-U24-P8A ~ R4-U1-P3A R4-U24-P8B ~ R4-U1-P4A 3 R4 U24 1 R4-U24-P10B ~ R1-U1-P8A 10 R4 U24 2 R4-U24-P11A ~ R2-U1-P9A R4 U24 3 R4-U24-P11B ~ R3-U1-P10A R4 U24 5 R4-U24-P9A ~ R5-U1-P5A R4 U24 6 R4-U24-P9B ~ R6-U1-P6A R4 U24 7 R4-U24-P10A ~ R7-U1-P7A 10 4 R4 U18 R4-U18-P8A ~ R4-U1-P3B R4-U18-P8B ~ R4-U1-P4B 3 R4 U18 1 R4-U18-P10B ~ R1-U1-P8B 10 R4 U18 2 R4-U18-P11A ~ R2-U1-P9B R4 U18 3 R4-U18-P11B ~ R3-U1-P10B R4 U18 5 R4-U18-P9A ~ R5-U1-P5B R4 U18 6 R4-U18-P9B ~ R6-U1-P6B R4 U18 7 R4-U18-P10A ~ R7-U1-P7B 10 (R5-U1). 203

48 5 R5 U24 R5-U24-P8A ~ R5-U1-P3A R5-U24-P8B ~ R5-U1-P4A 3 R5 U24 1 R5-U24-P10A ~ R1-U1-P7A 10 R5 U24 2 R5-U24-P10B ~ R2-U1-P8A 10 R5 U24 3 R5-U24-P11A ~ R3-U1-P9A R5 U24 4 R5-U24-P11B ~ R4-U1-P10A R5 U24 6 R5-U24-P9A ~ R6-U1-P5A R5 U24 7 R5-U24-P9B ~ R7-U1-P6A 5 R5 U18 R5-U18-P8A ~ R5-U1-P3B R5-U18-P8B ~ R5-U1-P4B 3 R5 U18 1 R5-U18-P10A ~ R1-U1-P7B 10 R5 U18 2 R5-U24-P10B ~ R2-U1-P8B 10 R5 U18 3 R5-U18-P11A ~ R3-U1-P9B R5 U18 4 R5-U18-P11B ~ R4-U1-P10B R5 U18 6 R5-U18-P9A ~ R6-U1-P5B R5 U18 7 R5-U18-P9B ~ R7-U1-P6B (R6-U1). 49 6 R6 U24 R6-U24-P8A ~ R6-U1-P3A R6-U24-P8B ~ R6-U1-P4A 3 R6 U24 1 R6-U24-P9B ~ R1-U1-P6A 10 R6 U24 2 R6-U24-P10A ~ R2-U1-P7A 10 R6 U24 3 R6-U24-P10B ~ R3-U1-P8A R6 U24 4 R6-U24-P11A ~ R4-U1-P9A R6 U24 5 R6-U24-P11B ~ R5-U1-P10A 204 SPARC SuperClusterT4-4 2012 12

49 ( ) R6 U24 7 R6-U24-P9A ~ R7-U1-P5A 6 R6 U18 R6-U18-P8A ~ R6-U1-P3B R6-U18-P8B ~ R6-U1-P4B 3 R6 U18 1 R6-U18-P9B ~ R1-U1-P6B 10 R6 U18 2 R6-U24-P10A ~ R2-U1-P7B 10 R6 U18 3 R6-U18-P10B ~ R3-U1-P8B R6 U18 4 R6-U18-P11A ~ R4-U1-P9B R6 U18 5 R6-U18-P11B ~ R5-U1-P10B R6 U18 7 R6-U24-P9A ~ R7-U1-P5B (R7-U1). 50 7 R7 U24 R7-U24-P8A ~ R7-U1-P3A R7-U24-P8B ~ R7-U1-P4A 3 R7 U24 1 R7-U24-P9A ~ R1-U1-P5A 10 R7 U24 2 R7-U24-P9B ~ R2-U1-P6A 10 R7 U24 3 R7-U24-P10A ~ R3-U1-P7A 10 R7 U24 4 R7-U24-P10B ~ R4-U1-P8A 10 R7 U24 5 R7-U24-P11A ~ R5-U1-P9A R7 U24 6 R7-U24-P11B ~ R6-U1-P10A 7 R7 U18 R7-U18-P8A ~ R7-U1-P3B R7-U18-P8B ~ R7-U1-P4B 3 R7 U18 1 R7-U18-P9A ~ R1-U1-P5B 10 R7 U18 2 R7-U24-P9B ~ R2-U1-P6B 10 R7 U18 3 R7-U18-P10A ~ R3-U1-P7B 10 R7 U18 4 R7-U18-P10B ~ R4-U1-P8B 10 205

50 ( ) R7 U18 5 R7-U18-P11A ~ R5-U1-P9B R7 U18 6 R7-U24-P11B ~ R6-U1-P10B (R1-U1). 51 1 R1 U24 R1-U24-P8A ~ R1-U1-P3A 3 R1 U24 2 R1-U24-P8B ~ R2-U1-P4A R1 U24 3 R1-U24-P9A ~ R3-U1-P5A R1 U24 4 R1-U24-P9B ~ R4-U1-P6A 10 R1 U24 5 R1-U24-P10A ~ R5-U1-P7A 10 R1 U24 6 R1-U24-P10B ~ R6-U1-P8A 10 R1 U24 7 R1-U24-P11A ~ R7-U1-P9A 10 R1 U24 8 R1-U24-P11B ~ R8-U1-P10A 10 1 R1 U18 R1-U18-P8A ~ R1-U1-P3B 3 R1 U18 2 R1-U18-P8B ~ R2-U1-P4B R1 U18 3 R1-U18-P9A ~ R3-U1-P5B R1 U18 4 R1-U18-P9B ~ R4-U1-P6B 10 R1 U18 5 R1-U18-P10A ~ R5-U1-P7B 10 R1 U18 6 R1-U18-P10B ~ R6-U1-P8B 10 R1 U18 7 R1-U18-P11A ~ R7-U1-P8B 10 R1 U18 8 R1-U18-P11B ~ R8-U1-P10B 10 (R2-U1). 206 SPARC SuperClusterT4-4 2012 12

52 2 R2 U24 R2-U24-P8A ~ R2-U1-P3A 3 R2 U24 1 R2-U24-P11B ~ R1-U1-P10A R2 U24 3 R2-U24-P8B ~ R3-U1-P4A R2 U24 4 R2-U24-P9A ~ R4-U1-P5A R2 U24 5 R2-U24-P9B ~ R5-U1-P6A 10 R2 U24 6 R2-U24-P10A ~ R6-U1-P7A 10 R2 U24 7 R2-U24-P10B ~ R7-U1-P8A 10 R2 U24 8 R2-U24-P11A ~ R8-U1-P9A 10 2 R2 U18 R2-U18-P8A ~ R2-U1-P3B 3 R2 U18 1 R2-U18-P11B ~ R1-U1-P10B R2 U18 3 R2-U18-P8B ~ R3-U1-P4B R2 U18 4 R2-U18-P9A ~ R4-U1-P5B R2 U18 5 R2-U18-P9B ~ R5-U1-P6B 10 R2 U18 6 R2-U18-P10A ~ R6-U1-P7B 10 R2 U18 7 R2-U18-P10B ~ R7-U1-P8B 10 R2 U18 8 R2-U18-P11A ~ R8-U1-P9B 10 (R3-U1). 53 3 R3 U24 R3-U24-P8A ~ R3-U1-P3A 3 R3 U24 1 R3-U24-P11A ~ R1-U1-P9A R3 U24 2 R3-U24-P11B ~ R2-U1-P10A R3 U24 4 R3-U24-P8B ~ R4-U1-P4A R3 U24 5 R3-U24-P9A ~ R5-U1-P5A R3 U24 6 R3-U24-P9B ~ R6-U1-P6A 207

53 ( ) R3 U24 7 R3-U24-P10A ~ R7-U1-P7A 10 R3 U24 8 R3-U24-P10B ~ R8-U1-P8A 10 3 R3 U18 R3-U18-P8A ~ R3-U1-P3B 3 R3 U18 1 R3-U18-P11A ~ R1-U1-P9B R3 U18 2 R3-U18-P11B ~ R2-U1-P10B R3 U18 4 R3-U18-P8B ~ R4-U1-P4B R3 U18 5 R3-U18-P9A ~ R5-U1-P5B R3 U18 6 R3-U18-P9B ~ R6-U1-P6B R3 U18 7 R3-U18-P10A ~ R7-U1-P7B 10 R3 U18 8 R3-U18-P10B ~ R8-U1-P8B 10 (R4-U1). 54 4 R4 U24 R4-U24-P8A ~ R4-U1-P3A 3 R4 U24 1 R4-U24-P10B ~ R1-U1-P8A 10 R4 U24 2 R4-U24-P11A ~ R2-U1-P9A R4 U24 3 R4-U24-P11B ~ R3-U1-P10A R4 U24 5 R4-U24-P8B ~ R5-U1-P4A R4 U24 6 R4-U24-P9A ~ R6-U1-P5A R4 U24 7 R4-U24-P9B ~ R7-U1-P6A 10 R4 U24 8 R4-U24-P10A ~ R8-U1-P7A 10 4 R4 U18 R4-U18-P8A ~ R4-U1-P3B 3 R4 U18 1 R4-U18-P10B ~ R1-U1-P8B 10 R4 U18 2 R4-U18-P11A ~ R2-U1-P9B R4 U18 3 R4-U18-P11B ~ R3-U1-P10B R4 U18 5 R4-U18-P8B ~ R5-U1-P4B 208 SPARC SuperClusterT4-4 2012 12

54 ( ) R4 U18 6 R4-U18-P9A ~ R6-U1-P5B R4 U18 7 R4-U18-P9B ~ R7-U1-P6B 10 R4 U18 8 R4-U18-P10A ~ R8-U1-P7B 10 (R5-U1). 55 5 R5 U24 R5-U24-P8A ~ R5-U1-P3A 3 R5 U24 1 R5-U24-P10A ~ R1-U1-P7A 10 R5 U24 2 R5-U24-P10B ~ R2-U1-P8A 10 R5 U24 3 R5-U24-P11A ~ R3-U1-P9A R5 U24 4 R5-U24-P11B ~ R4-U1-P10A R5 U24 6 R5-U24-P8B ~ R6-U1-P4A R5 U24 7 R5-U24-P9A ~ R7-U1-P5A R5 U24 8 R5-U24-P9B ~ R8-U1-P6A 10 5 R5 U18 R5-U18-P8A ~ R5-U1-P3B 3 R5 U18 1 R5-U18-P10A ~ R1-U1-P7B 10 R5 U18 2 R5-U24-P10B ~ R2-U1-P8B 10 R5 U18 3 R5-U18-P11A ~ R3-U1-P9B R5 U18 4 R5-U18-P11B ~ R4-U1-P10B R5 U18 6 R5-U18-P8B ~ R6-U1-P4B R5 U18 7 R5-U18-P9A ~ R7-U1-P5B R5 U18 8 R5-U18-P9B ~ R8-U1-P6B 10 (R6-U1). 209

56 6 R6 U24 R6-U24-P8A ~ R6-U1-P3A 3 R6 U24 1 R6-U24-P9B ~ R1-U1-P6A 10 R6 U24 2 R6-U24-P10A ~ R2-U1-P7A 10 R6 U24 3 R6-U24-P10B ~ R3-U1-P8A R6 U24 4 R6-U24-P11A ~ R4-U1-P9A R6 U24 5 R6-U24-P11B ~ R5-U1-P10A R6 U24 7 R6-U24-P8B ~ R7-U1-P4A R6 U24 8 R6-U24-P9A ~ R8-U1-P5A 6 R6 U18 R6-U18-P8A ~ R6-U1-P3B 3 R6 U18 1 R6-U18-P9B ~ R1-U1-P6B 10 R6 U18 2 R6-U24-P10A ~ R2-U1-P7B 10 R6 U18 3 R6-U18-P10B ~ R3-U1-P8B R6 U18 4 R6-U18-P11A ~ R4-U1-P9B R6 U18 5 R6-U18-P11B ~ R5-U1-P10B R6 U18 7 R6-U24-P8B ~ R7-U1-P4B R6 U18 8 R6-U18-P9A ~ R8-U1-P5B (R7-U1). 57 7 R7 U24 R7-U24-P8A ~ R7-U1-P3A 3 R7 U24 1 R7-U24-P9A ~ R1-U1-P5A 10 R7 U24 2 R7-U24-P9B ~ R2-U1-P6A 10 R7 U24 3 R7-U24-P10A ~ R3-U1-P7A 10 R7 U24 4 R7-U24-P10B ~ R4-U1-P8A 10 R7 U24 5 R7-U24-P11A ~ R5-U1-P9A 210 SPARC SuperClusterT4-4 2012 12

57 ( ) R7 U24 6 R7-U24-P11B ~ R6-U1-P10A R7 U24 8 R7-U24-P8B ~ R8-U1-P4A 7 R7 U18 R7-U18-P8A ~ R7-U1-P3B 3 R7 U18 1 R7-U18-P9A ~ R1-U1-P5B 10 R7 U18 2 R7-U24-P9B ~ R2-U1-P6B 10 R7 U18 3 R7-U18-P10A ~ R3-U1-P7B 10 R7 U18 4 R7-U18-P10B ~ R4-U1-P8B 10 R7 U18 5 R7-U18-P11A ~ R5-U1-P9B R7 U18 6 R7-U24-P11B ~ R6-U1-P10B R7 U18 8 R7-U24-P8B ~ R8-U1-P4B (R8-U1). 58 8 R8 U24 R8-U24-P8A ~ R8-U1-P3A 3 R8 U24 1 R8-U24-P8B ~ R1-U1-P4A 10 R8 U24 2 R8-U24-P9A ~ R2-U1-P5A 10 R8 U24 3 R8-U24-P9B ~ R3-U1-P6A 10 R8 U24 4 R8-U24-P10A ~ R4-U1-P7A 10 R8 U24 5 R8-U24-P10B ~ R5-U1-P8A R8 U24 6 R8-U24-P11A ~ R6-U1-P9A R8 U24 7 R8-U24-P11B ~ R7-U1-P10A 8 R8 U18 R8-U18-P8A ~ R8-U1-P3B 3 R8 U18 1 R8-U18-P8B ~ R1-U1-P4B 10 R8 U18 2 R8-U24-P9A ~ R2-U1-P5B 10 R8 U18 3 R8-U18-P9B ~ R3-U1-P6B 10 R8 U18 4 R8-U18-P10A ~ R4-U1-P7B 10 211

58 ( ) R8 U18 5 R8-U18-P10B ~ R5-U1-P8B R8 U18 6 R8-U24-P11A ~ R6-U1-P9B R8 U18 7 R8-U24-P1B ~ R7-U1-P10B 212 SPARC SuperClusterT4-4 2012 12

SPARC SuperCluster T4-4 Oracle Exadata Storage. 213 Oracle Exadata Storage 214 225 Oracle Exadata Storage 225 IP 226 ( ) 244 SPARC SuperCluster T4-4 Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage SPARC SuperCluster T4-4.,. Oracle Exadata Storage SPARC SuperCluster T4-4. Exadata Storage Server Oracle ASM(Oracle Automatic Storage Management). Oracle Exadata Storage. Oracle Exadata Storage SPARC SuperCluster T4-4. Oracle Exadata Storage,. Oracle Exadata Storage. 213

59 Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage 18 Exadata Storage Server, 600GB 15K RPM SAS 3TB 7.2K RPM SAS 1 3 Sun Datacenter InfiniBand Switch 36 6.75TB, (KVM) 15kVA PDU( 3, )2 48 Cisco Catalyst 4948 1, WS-C4948-S 1 2TB. 9 Exadata Storage Server, 600GB 15K RPM SAS 3TB 7.2K RPM SAS * 3 Sun Datacenter InfiniBand Switch 36 3.4TB, (KVM) 15kVA PDU( 3, )2 48 Cisco Catalyst 4948 1, WS-C4948-S 4 Exadata Storage Server, 600GB 15K RPM SAS 3TB 7.2K RPM SAS * 2 Sun Datacenter InfiniBand Switch 36 1.5TB, (KVM) 15kVA PDU( 3, )2 48 Cisco Catalyst 4948 1, WS-C4948-S Oracle Exadata Storage. SPARC SuperCluster T4-4., 57. 214 215 219 222 223 215 58 59 214 SPARC SuperClusterT4-4 2012 12

60 Exadata 1998mm 78.66 600mm 23.62 1200mm 47.24 1112mm 43.78 2300mm 90 914mm 36 ( ) 917.6kg 2023 ( ) 578.3kg 127 ( ) 396.8kg 87 58 59 215 60 61 216 PDU 61 Oracle Exadata Storage 12.6kW 6.9kW 3.4kW 12.9kVA 7.1kVA 3.5kVA 8.8kW 4.8kW 2.4kW 9.0kVA 5.0kVA 2.5kVA 215

PDU 216 PDU 61 60 PDU Oracle Exadata Storage. 3 Oracle. Oracle Exadata Storage 2 (Power Distribution Unit, PDU). 2 PDU. 62 PDU 1 63 3 64 1 65 3 66 63 Oracle Exadata Storage PDU kva 15kVA 1ph 200 ~ 240VAC PDU Amp 72A(3 x 24A) 42 C13 6 C19 3 216 SPARC SuperClusterT4-4 2012 12

PDU 63 Oracle Exadata Storage PDU ( ) 24A ( ) NEMA 30A/250VAC 2 /3 L6-30P PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 64 Oracle Exadata Storage 3 PDU 3 kva 14.4kVA 3ph 190 ~ 220VAC PDU Amp 69A(3 x 23A) 42 C13 6 C19 2 x 60A, 3ph 40A ( ) IEC 60309 60A 4 250VAC 3ph IP 67 PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 65 Oracle Exadata Storage PDU kva 15kVA 1ph 217

PDU 65 Oracle Exadata Storage PDU ( ) 220 ~ 240VAC PDU Amp 72A(3 x 24A) 42 C13 6 6 C19 25A ( ) IEC 60309 32A 3 250VAC IP44 PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 66 Oracle Exadata Storage 3 PDU kva 14.4kVA 3ph 220/380 ~ 240/415VAC PDU Amp 62.7A(3 x 20.9 A) 42 C13 6 C19 2 x 25A 25A ( ) IEC 60309 32A, 5 230/400VAC PDU 6 PDU 2m(6.6 ) PDU 4m(13 ). 218 SPARC SuperClusterT4-4 2012 12

60 61 215 219 219 222,. 67, 5 ~ 32 C (41 ~ 89.6 F) -40 ~ 70 C (-40 ~ 158 F) 21 ~ 23 C (70 ~ 47 F). 10 ~ 90%, 93% 45 ~ 50%,. 3048m(10000 ) 12000m(40000 ) 900m 300m 1. 219 222.. 68 BTU/ kj/ 43,000 45,400 30,100 31,800 BTU/ kj/ 23,600 24,900 16,500 17,400 219

68 ( ) BTU/ kj/ 11,600 12,250 8,100 8,600.. 914mm(36 ), 914mm(36 )... 220 SPARC SuperClusterT4-4 2012 12

18 69 ( ) CFM 2,000 1,390 CFM 1,090 760 CFM 530 375 219 222 221

. 400CFM... 70 4 3 2 219 219 222 68 68 69 71 Exadata 2159mm 8 1219mm 48 1575mm 62 ( ) 1001.1kg 2207 ( ) 659.6kg 1454 ( ) 475.3kg 1048 222 SPARC SuperClusterT4-4 2012 12

68 68 69 71 DNS 223 224 IP Oracle Exadata Storage Exadata Storage Server.. Exadata Storage Server. 1 (NET0) InfiniBand Sun QDR InfiniBand PCIe (Host Channel Adapter, HCA) 1 Oracle ILOM (NET MGT) 1 Oracle Exadata Storage Cisco Catalyst 4948. IP. IP NTP Oracle Exadata Storage. Oracle Exadata Storage 2... 223

IP Exadata Storage Server.,, Oracle ILOM.. PDU. Exadata Storage Server 2. 1GbE Oracle ILOM Oracle Integrated Lights Out Manager. InfiniBand InfiniBand Exadata Storage Server. Oracle Exadata Storage... 224 IP 71 DNS IP,. Oracle Exadata Storage. 72 Oracle Exadata Storage IP 42 18 (Exadata Storage Server 1 ) Oracle ILOM 18 (Exadata Storage Server 1 ) 4 (InfiniBand 3 1 ) PDU 2 (PDU ) 224 SPARC SuperClusterT4-4 2012 12

IP 72 Oracle Exadata Storage IP ( ) 24 9 (Exadata Storage Server 1 ) Oracle ILOM 9 (Exadata Storage Server 1 ) 4 (InfiniBand 3 1 ) PDU 2 (PDU ) 13 4 (Exadata Storage Server 1 ) Oracle ILOM 4 (Exadata Storage Server 1 ) 3 (InfiniBand 2, 1 ) PDU 2 (PDU ) 71 DNS 223 225 IP 226 ( ) Oracle Exadata Storage Oracle Exadata Storage SPARC SuperCluster T4-4. 73. 10GbE Oracle Exadata Storage. IP 73 Oracle Exadata Storage IP NET0 IP Oracle ILOM IP InfiniBand IP Exadata Storage Server 18 192.168.1.68 192.168.1.168 192.168.10.68 Exadata Storage Server 17 192.168.1.67 192.168.1.167 192.168.10.67 Exadata Storage Server 16 192.168.1.66 192.168.1.166 192.168.10.66 Exadata Storage Server 15 192.168.1.65 192.168.1.165 192.168.10.65 225

( ) 73 Oracle Exadata Storage IP ( ) NET0 IP Oracle ILOM IP InfiniBand IP Exadata Storage Server 14 192.168.1.64 192.168.1.164 192.168.10.64 Exadata Storage Server 13 192.168.1.63 192.168.1.163 192.168.10.63 Exadata Storage Server 12 192.168.1.62 192.168.1.162 192.168.10.62 Exadata Storage Server 11 192.168.1.61 192.168.1.161 192.168.10.61 Exadata Storage Server 10 192.168.1.60 192.168.1.160 192.168.10.60 Exadata Storage Server 9 192.168.1.59 192.168.1.159 192.168.10.59 Exadata Storage Server 8 192.168.1.58 192.168.1.158 192.168.10.58 Exadata Storage Server 7 192.168.1.57 192.168.1.157 192.168.10.57 Exadata Storage Server 6 192.168.1.56 192.168.1.156 192.168.10.56 Exadata Storage Server 5 192.168.1.55 192.168.1.155 192.168.10.55 Exadata Storage Server 4 192.168.1.54 192.168.1.154 192.168.10.54 Exadata Storage Server 3 192.168.1.53 192.168.1.153 192.168.10.53 Exadata Storage Server 2 192.168.1.52 192.168.1.152 192.168.10.52 Exadata Storage Server 1 192.168.1.51 192.168.1.151 192.168.10.51 Sun Datacenter InfiniBand Switch 36 3 192.168.1.223 Sun Datacenter InfiniBand Switch 36 2 192.168.1.222 Sun Datacenter InfiniBand Switch 36 1 192.168.1.221 192.168.1.220 PDU-A 192.168.1.212 PDU-B 192.168.1.213 ( ) Oracle Exadata Storage. 227 Oracle Exadata Storage 231 Oracle ILOM 232 235 237 3 239 InfiniBand 226 SPARC SuperClusterT4-4 2012 12

Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage.. 227

Oracle Exadata Storage 19 Oracle Exadata Storage Oracle Exadata Storage.. 228 SPARC SuperClusterT4-4 2012 12

Oracle Exadata Storage 20 Oracle Exadata Storage * U21 * * * Oracle Exadata Storage.. 229

Oracle Exadata Storage 21 Oracle Exadata Storage 230 SPARC SuperClusterT4-4 2012 12

Oracle ILOM Oracle ILOM Oracle ILOM. Oracle ILOM NET MGT Oracle Exadata Storage 21. Oracle Exadata Storage Oracle ILOM. 74 Oracle Exadata Storage Oracle ILOM U41 Exadata Storage Server 2 U39 Exadata Storage Server 4 U37 Exadata Storage Server 6 U35 Exadata Storage Server 8 U33 Exadata Storage Server 10 U31 Exadata Storage Server 12 U29 Exadata Storage Server 14 U27 Exadata Storage Server 18 U25 Exadata Storage Server 22 U18 Exadata Storage Server 26 U16 Exadata Storage Server 30 U14 Exadata Storage Server 32 U12 Exadata Storage Server 34 U10 Exadata Storage Server 36 U8 Exadata Storage Server 38 U6 Exadata Storage Server 40 U4 Exadata Storage Server 42 U2 Exadata Storage Server 44 Oracle Exadata Storage Oracle ILOM. 231

75 Oracle Exadata Storage Oracle ILOM U18 Exadata Storage Server 26 U16 Exadata Storage Server 30 U14 Exadata Storage Server 32 U12 Exadata Storage Server 34 U10 Exadata Storage Server 36 U8 Exadata Storage Server 38 U6 Exadata Storage Server 40 U4 Exadata Storage Server 42 U2 Exadata Storage Server 44 Oracle Exadata Storage Oracle ILOM. 76 Oracle Exadata Storage Oracle ILOM U8 Exadata Storage Server 38 U6 Exadata Storage Server 40 U4 Exadata Storage Server 42 U2 Exadata Storage Server 44. Net-0 Oracle Exadata Storage 21. Oracle Exadata Storage. 77 Oracle Exadata Storage U41 Exadata Storage Server 1 232 SPARC SuperClusterT4-4 2012 12

77 Oracle Exadata Storage ( ) U39 Exadata Storage Server 3 U37 Exadata Storage Server 5 U35 Exadata Storage Server 7 U33 Exadata Storage Server 9 U31 Exadata Storage Server 11 U29 Exadata Storage Server 13 U27 Exadata Storage Server 17 U25 Exadata Storage Server 21 U24 Sun Datacenter InfiniBand Switch 36 45 U20 Sun Datacenter InfiniBand Switch 36 46 U18 Exadata Storage Server 25 U16 Exadata Storage Server 29 U14 Exadata Storage Server 31 U12 Exadata Storage Server 33 U10 Exadata Storage Server 35 U8 Exadata Storage Server 37 U6 Exadata Storage Server 39 U4 Exadata Storage Server 41 U2 Exadata Storage Server 43 U1 Sun Datacenter InfiniBand Switch 36 47 PDU-A PDU 15 PDU-B PDU 19 Oracle Exadata Storage. 78 Oracle Exadata Storage U24 Sun Datacenter InfiniBand Switch 36 45 233

78 Oracle Exadata Storage ( ) U20 Sun Datacenter InfiniBand Switch 36 46 U18 Exadata Storage Server 25 U16 Exadata Storage Server 29 U14 Exadata Storage Server 31 U12 Exadata Storage Server 33 U10 Exadata Storage Server 35 U8 Exadata Storage Server 37 U6 Exadata Storage Server 39 U4 Exadata Storage Server 41 U2 Exadata Storage Server 43 U1 Sun Datacenter InfiniBand Switch 36 47 PDU-A PDU 15 PDU-B PDU 19 Oracle Exadata Storage. 79 Oracle Exadata Storage U24 Sun Datacenter InfiniBand Switch 36 45 U20 Sun Datacenter InfiniBand Switch 36 46 U8 Exadata Storage Server 37 U6 Exadata Storage Server 39 U4 Exadata Storage Server 41 U2 Exadata Storage Server 43 PDU-A PDU 15 PDU-B PDU 19 234 SPARC SuperClusterT4-4 2012 12

(Power Distribution Unit, PDU). PDU-A CMA,4. PDU Oracle Exadata Storage. 80 Oracle Exadata Storage PDU PDU-A/PS-00 PDU-B/PS-01 U41 G5-6 G0-0 2 U39 G5-3 G0-3 2 U37 G5-0 G0-6 2 U35 G4-6 G1-0 2 U33 G4-4 G1-2 2 U31 G4-2 G1-4 2 U29 G3-6 G2-0 2 U27 G3-5 G2-1 2 U25 G3-3 G2-3 2 U24 G3-1 G2-5 2 U23 G3-0 U22 G2-5 G3-1 1 U21 G3-0 G2-6 2 U20 G2-4 G3-2 2 U18 G2-2 G3-4 2 U16 G1-6 G4-0 2 U14 G2-0 G3-6 2 U12 G1-4 G4-2 2 U10 G1-2 G4-4 2 U8 G1-0 G4-6 2 U6 G0-6 G5-0 2 235

80 Oracle Exadata Storage PDU ( ) PDU-A/PS-00 PDU-B/PS-01 U4 G0-4 G5-2 2 U2 G0-2 G5-4 2 U1 G0-0 G5-6 2 PDU Oracle Exadata Storage. 81 Oracle Exadata Storage PDU PDU-A/PS-00 PDU-B/PS-01 U24 G3-1 G2-5 2 U23 G3-0 U22 G2-5 G3-1 1 U21 G3-0 G2-6 2 U20 G2-4 G3-2 2 U18 G2-2 G3-4 2 U16 G1-6 G4-0 2 U14 G2-0 G3-6 2 U12 G1-4 G4-2 2 U10 G1-2 G4-4 2 U8 G1-0 G4-6 2 U6 G0-6 G5-0 2 U4 G0-4 G5-2 2 U2 G0-2 G5-4 2 PDU Oracle Exadata Storage. 82 Oracle Exadata Storage PDU PDU-A/PS-00 PDU-B/PS-01 U24 G3-1 G2-5 2 236 SPARC SuperClusterT4-4 2012 12

3 82 Oracle Exadata Storage PDU ( ) PDU-A/PS-00 PDU-B/PS-01 U23 G3-0 U22 G2-5 G3-1 1 U21 G3-0 G2-6 2 U20 G2-4 G3-2 2 U8 G1-0 G4-6 2 U6 G0-6 G5-0 2 U4 G0-4 G5-2 2 U2 G0-2 G5-4 2 3 (Power Distribution Unit, PDU) 3. PDU-A CMA,4. PDU Oracle Exadata Storage. 83 Oracle Exadata Storage 3 PDU PDU-A/PS-00 PDU-B/PS-01 U41 G5-6 G2-0 2 U39 G5-3 G2-3 2 U37 G5-0 G2-6 2 U35 G4-6 G1-0 2 U33 G4-4 G1-2 2 U31 G4-2 G1-4 2 U29 G3-6 G0-0 2 U27 G3-5 G0-1 2 U25 G3-3 G0-3 2 U24 G3-1 G0-5 2 U23 G5-0 237

3 83 Oracle Exadata Storage 3 PDU ( ) PDU-A/PS-00 PDU-B/PS-01 U22 G2-5 G5-1 1 U21 G3-0 G0-6 2 U20 G2-4 G5-2 2 U18 G2-2 G5-4 2 U16 G1-6 G4-0 2 U14 G2-0 G5-6 2 U12 G1-4 G4-2 2 U10 G1-2 G4-4 2 U8 G1-0 G4-6 2 U6 G0-6 G3-0 2 U4 G0-4 G3-2 2 U2 G0-2 G3-4 2 U1 G0-0 G3-6 2 PDU Oracle Exadata Storage. 84 Oracle Exadata Storage 3 PDU PDU-A/PS-00 PDU-B/PS-01 U24 G3-1 G0-5 2 U23 G5-0 U22 G2-5 G5-1 1 U21 G3-0 G0-6 2 U20 G2-4 G5-2 2 U18 G2-2 G5-4 2 U16 G1-6 G4-0 2 U14 G2-0 G5-6 2 U12 G1-4 G4-2 2 U10 G1-2 G4-4 2 238 SPARC SuperClusterT4-4 2012 12

InfiniBand 84 Oracle Exadata Storage 3 PDU ( ) PDU-A/PS-00 PDU-B/PS-01 U8 G1-0 G4-6 2 U6 G0-6 G3-0 2 U4 G0-4 G3-2 2 U2 G0-2 G3-4 2 U1 G0-0 G3-6 2 PDU Oracle Exadata Storage. 85 Oracle Exadata Storage 3 PDU PDU-A/PS-00 PDU-B/PS-01 U24 G3-1 G0-5 2 U23 G5-0 U22 G2-5 G5-1 1 U21 G3-0 G0-6 2 U20 G2-4 G5-2 2 U8 G1-0 G4-6 2 U6 G0-6 G3-0 2 U4 G0-4 G3-2 2 U2 G0-2 G3-4 2 InfiniBand InfiniBand. Sun Datacenter InfiniBand Switch 36 1, 20 24. Oracle Exadata Storage InfiniBand,. 86 Oracle Exadata Storage InfiniBand InfiniBand U24 0A U41 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand 239

InfiniBand 86 Oracle Exadata Storage InfiniBand ( ) InfiniBand U24 0B U39 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 1A U37 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 1B U35 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 2A U33 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 2B U31 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 3A U29 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U24 4A U27 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U24 5A U25 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U24 13A U18 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 14A U16 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 14B U14 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 15A U12 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 15B U10 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 16A U8 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 16B U6 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17A U4 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17B U2 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 0A U41 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 0B U39 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 1A U37 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 1B U35 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 2A U33 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 2B U31 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 3A U29 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 4A U27 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U20 5A U25 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U20 13A U18 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand 240 SPARC SuperClusterT4-4 2012 12

InfiniBand 86 Oracle Exadata Storage InfiniBand ( ) InfiniBand U20 14A U16 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U20 14B U14 Exadata Storage Server PCIe 3, P1 2 QDR InfiniBand U20 15A U12 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 15B U10 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 16A U8 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 16B U6 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17A U4 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17B U2 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 9B U24 Sun Datacenter InfiniBand Switch 36 U20 10B U24 Sun Datacenter InfiniBand Switch 36 U20 11B U24 Sun Datacenter InfiniBand Switch 36 U20 8A U24 Sun Datacenter InfiniBand Switch 36 U20 9A U24 Sun Datacenter InfiniBand Switch 36 U20 10A U24 Sun Datacenter InfiniBand Switch 36 U20 11A U24 Sun Datacenter InfiniBand Switch 36 U1 1B U20 Sun Datacenter InfiniBand Switch 36 U1 0B U24 Sun Datacenter InfiniBand Switch 36 9A 10A 11A 8A 9B 10B 11B 8B 8B 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 3 QDR InfiniBand 3 QDR InfiniBand Oracle Exadata Storage InfiniBand,. 241

InfiniBand 87 Oracle Exadata Storage InfiniBand InfiniBand U24 13A U18 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 14A U16 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 14B U14 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 15A U12 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 15B U10 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 16A U8 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 16B U6 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17A U4 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17B U2 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 13A U18 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U20 14A U16 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U20 14B U14 Exadata Storage Server PCIe 3, P1 2 QDR InfiniBand U20 15A U12 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 15B U10 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 16A U8 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 16B U6 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17A U4 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17B U2 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 9B U24 Sun Datacenter InfiniBand Switch 36 U20 10B U24 Sun Datacenter InfiniBand Switch 36 U20 11B U24 Sun Datacenter InfiniBand Switch 36 U20 8A U24 Sun Datacenter InfiniBand Switch 36 U20 9A U24 Sun Datacenter InfiniBand Switch 36 9A 10A 11A 8A 9B 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 242 SPARC SuperClusterT4-4 2012 12

InfiniBand 87 Oracle Exadata Storage InfiniBand ( ) InfiniBand U20 10A U24 Sun Datacenter InfiniBand Switch 36 U20 11A U24 Sun Datacenter InfiniBand Switch 36 U1 1B U20 Sun Datacenter InfiniBand Switch 36 U1 0B U24 Sun Datacenter InfiniBand Switch 36 10B 11B 8B 8B 2 QDR InfiniBand 2 QDR InfiniBand 3 QDR InfiniBand 3 QDR InfiniBand Oracle Exadata Storage InfiniBand,. 88 Oracle Exadata Storage InfiniBand InfiniBand U24 16A U8 Exadata Storage Server PCIe 2, P2 2 QDR InfiniBand U24 16B U6 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17A U4 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U24 17B U2 Exadata Storage Server PCIe 3, P2 3 QDR InfiniBand U20 16A U8 Exadata Storage Server PCIe 2, P1 2 QDR InfiniBand U20 16B U6 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17A U4 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 17B U2 Exadata Storage Server PCIe 3, P1 3 QDR InfiniBand U20 9B U24 Sun Datacenter InfiniBand Switch 36 U20 10B U24 Sun Datacenter InfiniBand Switch 36 U20 11B U24 Sun Datacenter InfiniBand Switch 36 U20 8A U24 Sun Datacenter InfiniBand Switch 36 U20 9A U24 Sun Datacenter InfiniBand Switch 36 9A 10A 11A 8A 9B 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 2 QDR InfiniBand 243

SPARC SuperClusterT4-4 Oracle Exadata Storage 88 Oracle Exadata Storage InfiniBand ( ) InfiniBand U20 10A U24 Sun Datacenter InfiniBand Switch 36 U20 11A U24 Sun Datacenter InfiniBand Switch 36 10B 11B 2 QDR InfiniBand 2 QDR InfiniBand SPARC SuperClusterT4-4 Oracle Exadata Storage Sun Datacenter InfiniBand Switch 36. SPARC SuperCluster T4-4 Oracle Exadata Storage Oracle Exadata Storage Sun Datacenter InfiniBand Switch 36.,. InfiniBand QSFP(Quad Small Form-factor Pluggable)... Oracle Exadata Storage Sun Datacenter InfiniBand Switch 36 ( ) Oracle Exadata Storage. 245 SPARC SuperCluster T4-4 Oracle Exadata Storage. SPARC SuperClusterT4-4 InfiniBand SPARC SuperCluster T4-4 1(U1). 18(U18) 24(U24). 7... 244 SPARC SuperClusterT4-4 2012 12

SPARC SuperClusterT4-4 Oracle Exadata Storage Oracle Exadata Storage InfiniBand Oracle Exadata Storage 1(U1) (, Oracle Exadata Storage ). 20(U20) 24(U24). SPARC SuperClusterT4-4 Oracle Exadata Storage Oracle Exadata Storage Oracle SPARC SuperCluster T4-4. Oracle Exadata Storage SPARC SuperCluster T4-4. Oracle Exadata Storage Oracle Exadata Storage SPARC SuperCluster T4-4 248 SPARC SuperCluster T4-4 Oracle Exadata Storage Oracle Exadata Storage. 245

SPARC SuperClusterT4-4 Oracle Exadata Storage Oracle Exadata Storage SPARC SuperCluster T4-4. 7.. Oracle Exadata Storage 2. Oracle Exadata Storage. SPARC SuperCluster T4-4 SPARC SuperCluster T4-4 Oracle Exadata Storage Oracle Exadata Storage. Oracle Exadata Storage. 4. 246 SPARC SuperClusterT4-4 2012 12

SPARC SuperClusterT4-4 Oracle Exadata Storage. 1(R1) SPARC SuperCluster T4-4. 2(R2) Oracle Exadata Storage. InfiniBand InfiniBand. InfiniBand 1(IB1) SPARC SuperCluster T4-4 U1. InfiniBand 2(IB2). SPARC SuperCluster T4-4 U18 Oracle Exadata Storage U20 InfiniBand 3(IB3) U24. Oracle Exadata Storage SPARC SuperCluster T4-4. 247

SPARC SuperClusterT4-4 Oracle Exadata Storage 89 R1-IB3 2 R1-IB2 2 R1-IB3-P2A ~ R2-IB3-P2A R1-IB3-P2B ~ R2-IB3-P2B R1-IB3-P7B ~ R2-IB2-P7B R1-IB3-P12A ~ R2-IB2-P12A R1-IB2-P7B ~ R2-IB3-P7B R1-IB2-P12A ~ R2-IB3-P12A R1-IB2-P2A ~ R2-IB2-P2A R1-IB2-P2B ~ R2-IB2-P2B SPARC SuperClusterT4-4 Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage Oracle Exadata Storage Oracle SPARC SuperCluster T4-4. Oracle Exadata Storage SPARC SuperCluster T4-4 245 SPARC SuperCluster T4-4 Oracle Exadata Storage. 7. 8.,. 248 SPARC SuperClusterT4-4 2012 12

SPARC SuperClusterT4-4 Oracle Exadata Storage, 1. 2 1. 8 2 8. 1(R1) SPARC SuperCluster T4-4. 2(R2) Oracle Exadata Storage. InfiniBand InfiniBand. InfiniBand 1(IB1) U1. InfiniBand 2(IB2). SPARC SuperCluster T4-4 U18 Oracle Exadata Storage U20 InfiniBand 3(IB3) U24. (R1-IB1). 249