3 IBM WebSphere User Conference WAS (e-mail : cjh@kr.ibm.com ) SWG WebSphere FTSS 2005. 6. 28
날로복잡해져가는 J2EE 환경에대한체계적인관리운영시스템구축을위해 WebSphere Application Server 가제공하는각종툴에대한 Guideline 필요. Request Metrics Thread Analyzer Memory dump diagnostic? PMI Request Metrics Tivoli Performance Viewer WSAM ( Websphere Studio Application Monitor) Tivoli Performance Advisor
J2EE application stack WebSphere Performance tooling J2EE Runtime Performance Advisor Tuning with recommendation 관리자, OS Performance Viewer Request Metrics Tivoli Performance Advisor System tools Resource-level statistics Transaction-level statistics OS/Network performance 개발자 Hardware
WAS 의실시간모니터링및장애진단툴 PMI Request Metrics Tivoli Performance Viewer Tivoli Performance Advisor Log Analyzer Thread Analyzer Memory dump diagnostic/heaproots GC Analyzer WebSphere Studio Appplication Monitor GUI WAS GUI metrics API. activity.log WAS GUI JVM GUI. JVM Heap ( ) HeapDump GUI JVM GC(Garbage Collection) GUI WAS WAS Admin Console WAS Admin Console WAS Admin Console WAS Admin Console GUI Application
WAS 의로그파일들 access_log plugin.log SystemOut.log SystemErr.log native_out.log native_err.log activity.log tranlog HeapDump javacore.txt FFDC Error WAS, Console Out, Exception Error Exception Error WAS OS output WAS OS error WAS Log Analyzer WAS JVM Heap Memory, HeapRoots JVM First Failure Data Captuer WAS,. IBM WAS, WAS,, WAS,, WAS, WAS, WAS,, Transaction auto recovery WAS Memory leak WAS, WAS,
Thread Analyzer? WAS X- ray., WAS. hang. WAS id shell kill -3 PID. WAS_HOME javacore.xxx.txt.. Thread Analyzer. Thread Analyzer http://www14.software.ibm.com/webapp/download/search.jsp?go=y&rs=thread
Heap Analyzer/ Memory Dump Diagnostic Heap Dump? JVM JVM Heap. Heap Dump. Heap Dump WAS overflow Heap WAS_HOME. Heap Dump Heaproots Heap Analyzer Heap Analyzer. Heap Dump File Format IBM Portable Heap Dump (.phd) format IBM text heap dump format HPROF heap dump format SVC Dumps (on IBM z-series) Heap Analyzer Download http://www.alphaworks.ibm.com/tech/heapanalyzer
Performance Monitoring Infrastructure(PMI) PMI. PMI.
Performance Monitoring Infrastructure(PMI): top 10 Metrics (Average response time) EJB (Number of requests) WAS capacity. HTTP (Number of live HTTP sessions) EJB.. JVM GC JVM heap. CPU I/O CPU I/O, paging workload capacity.
Tivoli Performance Viewer Tivoli Performance Viewer, GUI., Tivoli Performance Viewer,. Tivoli Performance Viewer 에의해모니터링가능한요소들 시스템자원들 - JDBC Connection Pool, Thread Pool, Dynamic Cache 등각종 Pool 과 Cache 운영상황, - Heap memory, GC, 자바객체의동향등 JVM 운영상황 - CPU 사용량등시스템운영상황 분산객체들 - EJB, Servlet, JSP 등의호출건수, 평균응답시간 ( 개별컴포넌트도모니터링가능, 예를들면특정 EJB 컴포넌트나서블릿페이지별로도모니터링가능 ) 분산서버들 - 클러스터및서버들의운영상황 특히, Tivoli performance viewer 에는장시간의성능기록을보관하고이를나중에재생함으로써시스템관리자가시스템의사용현황을쉽고빠르게 Review 할수있도록하는기능이포함되어있다. - Playback 기능은 1 배속 /5 배속 /20 배속 /60 배속등으로재생속도를달리할수있게되어있다. 이를이용하여관리자는장시간의성능데이터를빠르게분석할수있을뿐아니라관리자가화면을보고있지않은동안에발생한일들도추후검토할수있다.
Base TPV/PMI Overhead reqs/second 180 160 140 120 100 80 60 40 20 0 163 161 157 154 150 148 PMI only no TPV Basic Refresh 30 Buffer 40 (default) Refresh 10 Buffer 40 CPU% W AS 100 100 100 100 100 100 R/T ms 301 316 302 324 306 331 System configuration Driver: W orkload driver on IBM xseries 335, 2 x 2.8 GHz Xeon (HT enabled), 4GB RAM, Red Hat Enterprise Linux 3, 50 clients SUT: IBM xseries 335, 2 x 2.8 GHz Xeon, 4GB RAM, HT disabled, Windows 2003 Server Standard Edition Database: IBM xseries 335, 2 x 2.8 GHz Xeon (HT enabled), 4GB RAM, W indows 2003 Server Standard Edition, DB2 v8.1 fp6, JDBC Provider: DB2 Universal JCC DriverXA Type 4 All Tivoli Performance Viewer? Tivoli Performance Viewer Refresh 3%. Tivoli Performance Viewer Tivoli Performance Viewer.
Performance Advisor WebSphere Application Server WebSphere Application Server PMI. WebSphere Application Server... Different Types ORB service thread pools Web container thread pools Connection pool size Prepared statement cache size Session cache size Persisted session size and time Dynamic cache size JVM heap size DB2 Performance Configuration Wizard
Request Metrics. ARM :. IP URI EJB JMS
Log Analyzer Log Analyzer. Log Analyzer. Log Analyzer IBM. Log Analyzer. Java Thread Dump Active Java Thread Hang WAS_HOME/bin/waslogbr.sh activity.log
WebSphere Application Server Node: Intel XEON @ 2.7 GHz x 2 3.86 GB Memory 100 MB Ethernet Linux 2.4 kernel DB2 Node: Intel XEON @ 3.2GHz x 2 Two-way 2.8GHz CPU 3.86 GB Memory 100 MB Ethernet Linux 2.4 kernel Version: IBM DB2 V 8.2, DB2 Universal JCC Type 4 (XA) Driver Number of concurrent users: 50 Runtime 커넥션풀정보 Trade DataSource Connection Pool Size: 5/5 WebContainer Thread Pool Size: 50/50 PrepareStatement Cache Size: 60 TPS: 181 reqs/sec Runtime Performance Advisor, Tivoli Performance Viewer Connection pool size: 10 TPS: 254 reqs/sec 40 percents 성능증가
스레드풀에서의성능버틀넥 TradeDataSource Connection Pool Min Size: 1 Max Size : 10 WebContainer Thread Pool Min Size: 20 Max Size: 20 Concurrent users: 50 Prepared Statement Cache Size: 60 JVM Heap Size: 1024 CPU Usage during steady state (i.e., no load): ~59 percent Active Concurrent Thread Count: 20 Thread Pool Size: 20 JDBC wait time: 12 milliseconds. CPU Usage: 70 percent Throughput: 150 requests/sec TPV, PA Active Concurrent Thread Count: 30 Thread Pool Size: 50 JDBC wait time: 12 milliseconds CPU Usage: 70 percent Throughput: 257 request/sec.
Request metrics 를통한문제영역검출 Request metrics (Application: servlet/ejb) 1. TPV.(TPV summary response time ) 2. Request Metrics. 3.. 4. ARM agent SystemOut.log. 5. 4. Servlet /myapp/myservlet ------> 950 ms EJB UserAccount.getAccountData ------> 35 ms JDBC select ------> 7 ms findbyprimarykey 메소드를매 req 마다호출함으로인해 EJB 호출에드는비용이큼
부적절한캐쉬사이즈로인한성능 bottleneck Configuration: TradeDataSource Connection Pool Min Size: 1 Max Size: 10 WebContainer Thread Pool Min Size: 50 Max Size: 50 Concurrent Clients: 50 Prepared Statement Cache Size: 5 JVM Heap Size: 1024 Throughput: 226 reqs/sec. TPV, PA Prepared Statement Cache size: 20 TPS: 249 reqs/sec Too high discard rate
WSAM? J2EE. JVM -
WSAM J2EE -
WSAM
7. WSAM 이제공하는세가지관점에서의운영관리방안 자원가용성관리 (Interval Data) 장애문제해결 (Real-time Data) 성능측정및리포팅 (Historical) 장애발생시 Sampling 서버및자원상태실시간모니터링실시간가동중인애플리케이션모니터링서버간설치된바이너리비교기능서버간설치된런타임 / 구성환경비교기능서버자원비교분석기능 문제에대한스레드단위까지의추적기능애플리케이션수행흐름분석모니터링이상현상발생시 SNMP추적과추적로깅스레드제어및스레드덤프캡쳐애플리케이션 Deadlock 검출및분석 Memory leak 검출기능 수행빈도높은애플리케이션레포팅수행된 SQL 관련레포트기능 TOP 레포트기능서버자원활용률레포트기능
7.1. 자원가용성관리 3 IBM WebSphere User Conference WAS JVM, CPU.
7.1. 자원가용성관리 3 IBM WebSphere User Conference In-Flight WAS. WAS request, WAS. ID request
7.1. 자원가용성관리 3 IBM WebSphere User Conference. 서버간비교결과요약 서버간비교결과상세
7.1. 자원가용성관리 3 IBM WebSphere User Conference. 서버간런타임비교결과
7.1. 자원가용성관리 3 IBM WebSphere User Conference 2 60 48.
3 IBM WebSphere User Conference WAS WAS fail over WAS., WAS.
3 IBM WebSphere User Conference DB lock, DB lock WAS-DB. class method connect.
3 IBM WebSphere User Conference / CPU Time.
3 IBM WebSphere User Conference SNMP SNMP SMS. history,,. 메소드추적
3 IBM WebSphere User Conference JVM JVM.,.
3 IBM WebSphere User Conference Lock Synchronization call lock acquired, released.
3 IBM WebSphere User Conference Memory Leak leak Memory Leak Memory Leak Class
3 IBM WebSphere User Conference
3 IBM WebSphere User Conference. 수행빈도가높은어플리케이션정보 CPU Time 이높은순서의메소드관련정보 관련각어플리케이션의수행흐름정보
SQL SQL.
3 IBM WebSphere User Conference TOP, SQL Call, TOP. 가장느린 request 분석레포트 SQL call 이가장많은 request 분석레포트
3 IBM WebSphere User Conference WAS 서버의가동률레포트. WAS 서버간의가동률비교레포트
3 IBM WebSphere User Conference 2005 8 WAS hang WSAM Thread : 300 DataSource : 30
3 IBM WebSphere User Conference
3 IBM WebSphere User Conference
3 IBM WebSphere User Conference
3 IBM WebSphere User Conference 수정사항 Oracle DataBase 연결방식을 Type2(OCI 방식 ) 에서 Type4 로변경. JNDI lookup 시의코드를 reference 방식으로변경. 한 jsp 에서 datasource 를 2 번호출하는것을 1 번만호출하도록변경. 개선사항 변경전에는 active user 50 에서 hang 현상이발생하였으나변경후에는 Active User 230 이되어도정상적으로서비스. 변경전에비해개별적인 response 타임에는변화가없지만전체적인 TPS 가좋아짐 결과적으로예전에비해훨씬많은 transaction 을보장
자원가용성관리 (Interval Data) 장애문제해결 (Real-time Data) 성능측정및리포팅 (Historical) 장애발생시 Sampling OS 제공각종모니터링툴 Heap Analyzer Thread Analyzer Request Metrics Tivoli Performance Advisor Tivoli Performance Viewer WSAM ( Websphere Studio Application Monitor)