슬라이드 1 - PDF Free Download

Enterprise 환경에서 Tomcat 운영하기 Part 1 : Tomcat 잘알고사용하기 Victor Lee sleepred@gmail.com 2015.12.29

Tomcat 개요 Tomcat Architecture Basic Features - Configuration, Connector, Tomcat JDBC Connection Pool 등 Advance Features - 웹서버연동, Session Clustering 등 Management & Monitoring - Tomcat Manager, JK Manager, VisualVM, Scouter 등 Tomcat Troubleshooting & 운영노하우 Tomcat Community 소개 - Build, Tomcat 확장하기, Contribution

Complexity Vendor Dependency Cost Overrun Solution?

Solution? Stability Security Availability

Top Open Source Communities ASF : http://www.apache.org/

Apache Tomcat : http://tomcat.apache.org/ User Mailing list : http://mail-archives.apache.org/mod_mbox/tomcat-users/

https://plumbr.eu/blog/java/most-popular-java-ee-containers-2015-edition

WEBSITE LISTS AVAILABLE Create a list of 158,040 websites using Apache Tomcat. https://wappalyzer.com/applications/apache-tomcat

1996 년 6 월제 4 회 JavaOne ( 당시 Sun Microsystems 주최 ) 가개발하고있던, JSWDK(JavaServer Web Development Kit) 을 Apache Software Foundation 에기증하여시작 되었음. Tomcat 은 Apache Software Foundation(ASF) 하위의 Apache Tomcat 프로젝트에서가개발되고있는 Sevlet/JSP Container 가포함된웹어플리케이션서버 (WAS) Tomcat History Tomcat 3.x (1999년) : 최초릴리즈 Tomcat 4.x (2003년) : Catalina 아키텍처도입 Tomcat 5.x (2004년) : Session Replication 기능구현 Tomcat 6.0 (2007년) : Primary-Secondary 방식 Sessioin Replication 구현 Tomcat 7.0 (2011년) : Security 강화, Connector 리팩토링 Tomcat 8.0 (2014년) : Resouce 리팩토링, WebSocket 기능추가 Tomcat 9.0 (2015년) : 9.0.0.M1(alpha) Release (2015-11-19)

Tomcat 은 Serlvet Spec 별 Major 버전으로릴리즈 Servlet 버전과 Tomcat 사양 (Spec) Servlet Spec JSP Spec EL Spec WebSocket Spec Tomcat version Release revision Support Java Versions 4.0 TBD (2.4?) TBD (3.1?) TBD (1.2?) 9.0.x 9.0.0.M1 (alpha) 8 and later 3.1 2.3 3.0 1.1 8.0.x 8.0.28 7 and later 3.0 2.2 2.2 1.1 7.0.x 7.0.65 6 and later (WebSocket 1.1 requires 7 or later) 2.5 2.1 2.1 N/A 6.0.x 6.0.44 5 and later 2.4 2.0 N/A N/A 5.5.x (archived) 5.5.36 (archived) 1.4 and later 2.3 1.2 N/A N/A 4.1.x (archived) 4.1.40 (archived) 1.3 and later 2.2 1.1 N/A N/A 3.3.x (archived) 3.3.2 (archived) 1.1 and later TBD : To Be Determined Community support version : Tomcat6, Tomcat7, Tomcat8, Tomcat9 End of life for Apache Tomcat 6.0.x : 31 December 2016.

Fail-over Fail-over Load Balancing Session Server Load Balancing WEB WAS Session replication Browser(User) Network L4 / Switch DB OpenEJB

Coyote Jasper Catalina

Server Service(*) Engine Host(*) Context(*) Connector Connector Connector Valve Valve Valve Valve Valve Valve Global Resource Clustering Support JMX Support JSP Support Security Utility R e q u e s t Realm Realm Realm Wrapper(*) Valve Valve Valve Listener Valve Listener Valve Listener Valve Listener Logger Resources Executor Loader Manager Listener Listener Listener Valve Listener Jar Scanner Resource Resource JSP JSP Servlets Servlet

Server : Server = Catalina servlet container = Tomcat Instance JVM 안에서 Singleton 으로존재, Server 내에포함되는 Service 들 Life Cycle 관리담당 Context : web application 을나타내는구성요소, Spec 상 ServletContext 와맵핑. WAR/EAR 파일이나, exploded 된디렉토리 http://tomcat.apache.org/tomcat-8.0-doc/config/server.html 를참고

Listeners : Tomcat Lifecycle 이벤트발생시 action 을수행하는컴포넌트 Tomcat 의 lifecycle 을커스터마이징하고싶은경우활용가능 ex) APR Lifecycle Listener, Global Resources Lifecycle Listener 등 Manager : HTTP 세션을생성관리해주는 session manager Logger : 안에 JULI(Java Logging implimentation) 라는자체구현체를제공

Valve : 컨테이너 (Engine, Host, Context) 와관련된각각의 request 처리 pipeline(valve chain 객체 ) 안에들어가있는컴포넌트

Server.xml 예제 <?xml version='1.0' encoding='utf-8'?> <Server port="8005" shutdown="shutdown"> <Listener classname="org.apache.catalina.mbeans.globalresourceslifecyclelistener" /> <GlobalNamingResources> <Resource name="userdatabase" auth="container" type="org.apache.catalina.userdatabase" description="user database that can be updated and saved" factory="org.apache.catalina.users.memoryuserdatabasefactory" pathname="conf/tomcat-users.xml" /> </GlobalNamingResources> <Service name="catalina"> <Connector port="8080" protocol="http/1.1" connectiontimeout="20000" redirectport="8443" /> <Engine name="catalina" defaulthost="localhost"> <Realm classname="org.apache.catalina.realm.userdatabaserealm" resourcename="userdatabase"/> <Host name="localhost" appbase="webapps" unpackwars="true" autodeploy="true" xmlvalidation="false" xmlnamespaceaware="false"> <Valve classname="org.apache.catalina.valves.accesslogvalve" directory="logs" prefix="localhost_access_log" suffix=".txt" pattern="%h %l %u %t "%r" %s %b" /> </Host> </Engine> </Service> </Server> It is NOT recommended to place <Context> elements directly in the server.xml file (https://tomcat.apache.org/tomcat-8.0-doc/config/context.html)

Tomcat Client 로부터요청을받아 Container 에전달하기위한 Component Connector 는 Tomcat 의웹서버기능으로 Coyote Architecture 로구현됨 Protocal 과처리방식에따른구현체를가짐 Connector Protocal HTTP/HTTPS (HTTP/1.1) AJP (AJP/1.3) Connector 처리방식 Java Blocking I/O (BIO) : All I/O operations are blocking in processor thread (Read, Write, etc. ) Simple, stable, mature Java Non-blocking I/O (NIO) : Allows huge number of parallel requests Java NIO.2 (NIO2) : like the NIO connector but uses the NIO2 framework Native / Apache Portable Runtime (APR) : Native code (C), Use JNI http://www.tecbar.net/optimize-tomcat-performance/

Tomcat Connector 성능 이론적으로는 APR > NIO > BIO 이나, JAVA 버전, OS 설정, 사용자부하유형에따라상이함 Stability : BIO > NIO or APR SSL : APR > NIO > BIO Low concurrency : BIO > APR > NIO High concurrency No Keep-Alive : BIO > APR > NIO High concurrency Keep-Alive : APR > NIO > BIO http://www.tecbar.net/optimize-tomcat-performance/

Tomcat HTTP Connector 는 HTTP/1.1 protocol 을지원하는컴포넌트 속성 : <Service name="catalina"> <Connector port="8080" protocol="http/1.1 connectiontimeout="20000 redirectport="8443" /> </Service> 항목설명기본값 protocol HTTP, AJP 등 Connector 의 Protocal 을설정하는항목으로구현체를직접설정도가능함 - BIO(java connector) : org.apache.coyote. http11.http11protocol - NIO(java connector) : org.apache.coyote. http11.http11nioprotocol - NIO2(java connector) : org.apache.coyote. http11.http11nio2protocol - APR(native connector) : org.apache.coyote. http11.http11aprprotocol HTTP/1.1

Tomcat HTTP Connector 는 HTTP/1.1 protocol 을지원하는컴포넌트 http://tomcat.apache.org/tomcat-8.0-doc/config/http.html http://tomcat.apache.org/tomcat-8.0-doc/config/http.html#connector_comparison

Tomcat AJP Connector 는 AJP protocol 을사용하여 Tomcat 과 web server 와통신할때사용되는컴포넌트 속성 : <Service name="catalina"> <Connector port="8089" protocol= AJP/1.3 redirectport="8443" /> </Service> 항목설명기본값 protocol HTTP, AJP 등 Connector 의 Protocal 을설정하는항목으로구현체를직접설정도가능함 - BIO : org.apache.coyote.ajp.ajpprotocol - NIO : org.apache.coyote.ajp.ajpnioprotocol - NIO2 : org.apache.coyote.ajp.ajpnio2protocol - APR : org.apache.coyote.ajp.ajpaprprotocol AJP/1.3

Tomcat AJP Connector 는 AJP protocol 을사용하여 Tomcat 과 web server 와통신할때사용되는컴포넌트 http://tomcat.apache.org/tomcat-8.0-doc/config/ajp.html http://tomcat.apache.org/tomcat-8.0-doc/config/http.html#connector_comparison http://www.tecbar.net/optimize-tomcat-performance/

Tomcat Logging JULI Logging Framework 사용 Apache Commons Logging 기반으로구현 java.util.logging 사용 (default) extra 패키지를통해 Log4j 등 Logging Framework 변경가능 JULI Logging Architecture

Tomcat Logging ClassLoaderLogManager java.util.logging.logmanager를확장 - $CATALINA_BASE/conf/logging.properties 파일 Loading 하여, Tomcat에적용 - ClassLoader의 path에서 logging.properties를찾아, ClassLoader별 Log 구성을적용 -> 웹어플리케이션별 log Configuration 구현 # ----------------------------------------------------------------------------- # LOGGING_MANAGER (Optional) Override Tomcat's logging manager # Example (all one line) # LOGGING_MANAGER="-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager" # ----------------------------------------------------------------------------- if [ -z "$LOGGING_MANAGER" ]; then LOGGING_MANAGER="-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager" fi

Tomcat Logging logging.properties handlers = 1catalina.org.apache.juli.AsyncFileHandler, 2localhost.org.apache.juli.AsyncFileHandler, java.util.logging.consolehandler.handlers = 1catalina.org.apache.juli.AsyncFileHandler, java.util.logging.consolehandler ############################################################ # Handler specific properties. # Describes specific configuration info for Handlers. ############################################################ 1catalina.org.apache.juli.AsyncFileHandler.level = FINE 1catalina.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs 1catalina.org.apache.juli.AsyncFileHandler.prefix = catalina. 2localhost.org.apache.juli.AsyncFileHandler.level = FINE 2localhost.org.apache.juli.AsyncFileHandler.directory = ${catalina.base}/logs 2localhost.org.apache.juli.AsyncFileHandler.prefix = localhost. ############################################################ # Facility specific properties. # Provides extra control for each logger. ############################################################ java.util.logging.consolehandler.level = FINE java.util.logging.consolehandler.formatter = org.apache.juli.onelineformatter org.apache.catalina.core.containerbase.[catalina].[localhost].level = INFO org.apache.catalina.core.containerbase.[catalina].[localhost].handlers = 2localhost.org.apache.juli.AsyncFileHandler

Tomcat Logging Tomcat Logging 종류 java.util.logging, javax.servlet.servletcontext.log, Console, Access Logging 이제공됨 구분설명파일명 Java Logging API Servlets logging API Tomcat 에서는 Application 당로깅설정을제어하기위해 java.util.logging 기반의구현체 JULI 를사용 logging.properties 파일의설정에따름 javax.servlet.servletcontext.log( ) 를통해메시지를출력할때사용됨 logging.properites 파일의설정중 org.apache.catalina.core.containerbase.[${engine}].[${host}].[${context}] 의설정에따름 ex) localhost.yyyy-mmdd.log ex) manager.yyyy-mmdd.log Console Tomcat 운영시출력되는 STDERR/STDOUT ex) catalina.out Access logging AccessLogValve 또는 ExtendedAccessLogValve 에서생성하는로그 Access Log Valve 의설정에따름 ex) access_log.yyyy-mmdd

Tomcat Session Manager Tomcat Session Manager HTTP 세션을생성관리해주는 session manager 로 Context 안에설정하며, 설정이안되어있는경 우자동으로기본설정값에의해생성됨 (default : org.apache.catalina.session.standardmanager) <Manager classname= org.apache.catalina.session.standardmanager /> Session Manager 종류 구분설명파일명 StandardManager PersistentManager 기본설정. 한개의 instance 를사용하는경우만적용가능 디스크또는 DB 에세션을 Persist. 세션스와핑과장애대처 (fault tolerance) org.apache.catalina.sess ion.standardmanager org.apache.catalina.sess ion.persistentmanager DeltaManager BackupManager All-to-All 방식 Session Replication 기능구현 Primary-Secondary Session Replication 기능을구현 org.apache.catalina.sess ion.deltamanager org.apache.catalina.sess ion.backupmanager http://tomcat.apache.org/tomcat-8.0-doc/config/manager.html 를참고

Tomcat Connnection Pool Tomcat 은 DataBase Connection Pool 로 Commons DBCP 와 Tomcat JDBC Pool 의 2 개 지 Pool 을제공 Commons DBCP (http://commons.apache.org/proper/commons -dbcp/) The Tomcat JDBC Connection Pool (https://tomcat.apache.org/tomcat-8.0-doc/jdbcpool.html) Tomcat 7.0.18 extra package 로포함 Tomcat 7.0.19 정식 Release

Tomcat JDBC Connnection Pool Tomcat 은 DataBase Connection Pool 로 Commons DBCP 와 Tomcat JDBC Pool 의 2 개지 Pool 을 제공 Commons DBCP : org.apache.tomcat.dbcp.dbcp.basicdatasourcefactory (default) - org.apache.commons -> org.apache.tomcat. 패키지변경 Tomcat JDBC : org.apache.tomcat.jdbc.pool.datasourcefactory DBCP 는 Apache Commons 프로젝트가제공하는, DataBase Connection Pool 로오픈소스 DataBase Connection Pool 를가장많이알려져있고, 많이사용하고있다. Tomcat JDBC 는 Apache Tomcat 프로젝트가독립적으로개발한 DataBase Connection Pool 이다. <Resource type="javax.sql.datasource" name="jdbc/testdb" factory="org.apache.tomcat.jdbc.pool.datasourcefactory" driverclassname="com.mysql.jdbc.driver" url="jdbc:mysql://localhost:3306/mysql" username="mysql_user" password="mypassword123" />

Tomcat JDBC Connnection Pool 기본설정 공통 DB 접속설정 구분설명비고 username password url driverclassname connectionproperties DB 연결을위해 JDBC 드라이버에전달되는사용자이름 DB 연결을위해 JDBC 드라이버에전달되는사용자패스워드 DB 연결을위해 JDBC 드라이버에전달되는연결 URL JDBC 드라이버의 FQCN (Fully Qualified Class Name) DB 연결을위해 JDBC 드라이버에전달되는연결 Properties 항목설명기본값 defaultautocommit 생성된 Connection 의자동 Commit Mode JDBC Dirver 기본값 - Oracle : true - MySQL/MariaDB : true defaultreadonly 생성된 Connection 의 ReadOnly JDBC Dirver 기본값 defaulttransactionisolation 생성된 Connection 의 TransactionIsolation JDBC Dirver 기본값 defaultcatalog 생성된 Connection 의 defaultcatalog JDBC Dirver 기본값 Commons DBCP : defaultautocommit 의기본값은 true

공통 DB 접속설정 항목 설명 기본값 initialsize Pool 기동시생성되는 Connection의초기사이즈 10 maxactive Pool에서동시에유지할수있는최대연결수 100 maxidle Pool에서유지하는최대 Connection 수 100 minidle Pool에서유지하는최소 Connection 수 initialsize maxwait 사용가능한 Connection이존재하지않는경우대기하는최대시간 30,000 (30초) maxage fairqueue Connection 을보존하는시간. Connection 반환시에 maxage 이상경과했으면, 해당 Connection 을 Close 시킴 getconnection 호출이 Connection 이 FIFO 방식으로공정하게제공하려면 true 로설정. org.apache.tomcat.jdbc.pool.fairblockingqueue 로구현되어있으며, 쓰레드가도착한순서대로 Connection 을획득하는것이보장됨 0 (msc) true

Tomcat JDBC Connnection Pool 기본설정 Connection Validation 관련설정 validationquery testonborrow 항목설명기본값 Connection 유효성확인을위한 SQL Query Pool 에서끄낸 Connection 을전달하기전에유효성을검증검증에실패했을경우, 해당연결을삭제하고, 새롭게 Pool 에서 Connection 을끄내려고시도 MySQL/MariaDB : SELECT 1 Oracle : SELECT 1 FROM DUAL False, true 설정시 validationquery 필요 testonreturn Pool 에 Connection 을반환하기전에 Connection 의유효성을검증 false, true 설정시 validationquery 필요 testwileidle Idle 상태의 Connection 에대한유효성을검증검증에실패할경우해당 Connection 을파기함이기능은 Evictor 를이용하기떄문에 Evictor 를활성화해야됨 (timebetweenevictionrunsmillis 설정참조 ) false, true 설정시 validationquery 필요 initsql validationclassname validationinterval Connection 을만들떄실행되는 SQL Query Validation Class 명을지정. 이속성을지정시 validationquery 에의한검증대신, Validator Class 에의한연결의검증이이루어짐 (org.apache.tomcat.jdbc.pool.validator 인터페이스구현필요 ) Validation 의실행간격지정. 마지막검증으로부터이값이상경과하지않은경우연결검증을거치지않음 30000 (30 초 ) logvalidationerrors Validation 중오류발생시, 로그출력할지에대한설정 false

PoolCleaner 관련설정 (Evictor) 항목 설명 기본값 timebetweenevictionrunsmilles Idle 상태의 Connection 을 check 하는간격, 5,000 (5초) minevictableidletimemillis Idle Connection의생존기간 60,000(60초 ) * Commons DBCP의속성중 numtestsperevictionrun 속성삭제 Connection Leak 검출설정 (removeabandoned) PoolCleaner 관련설정 (removeabandoned) 항목 설명 기본값 removeabandoned Connection 누수감지설정사용여부 false removeabandonedtimeout Connection 누수로판단하기까지의시간 60 초 logabandoned suspecttimeout abandonwhenpercentagefull Connection 누수를감지했을때, Connection 을 Close 하지않은어플리케이션의 Stuck Thread 를로그로출력여부의설정 Connection 누수의의심에대한판단할때까지의시간 suspecttimeout 시간을초과한 Connection 에대해경고로그및 JMX 에통지를수행 누수된 Connection 을파기할비율을설정 0~100 사이로설정하며, 이설정을초과한누수 Connection 을파기, removeabandonedtimeout 가초과한 Connection 은모두파기. false 0 0

DataSouce Resource 설정 <Resource name="${jdbc.resource.name}" factory= org.apache.tomcat.jdbc.pool.datasourcefactory" auth="container" type="javax.sql.datasource" username="${jdbc.username}" password="${jdbc.password}" driverclassname="${jdbc.driverclassname}" url="${jdbc.url}" testwhileidle="true" testonborrow="true" testonreturn="false" validationquery="select 1" validationinterval="30000" timebetweenevictionrunsmillis="5000" maxactive="100" minidle="10" maxwait="10000" initialsize="10" removeabandonedtimeout="60" removeabandoned="true" logabandoned="true" minevictableidletimemillis="30000" jmxenabled="true" jdbcinterceptors="connectionstate;statementfinalizer;slowqueryreportjmx(threshold=10000)"/>

Global Resource Level DataSource Pool Global level 공유 Global Resource(server.xml) + Resouce Link(application-context.xml) 해당 Application 에서만사용가능 Global Resource(server.xml) + Resouce Link(Global context.xml) 모든 Application 에서사용가능 Context Level DataSource Pool 해당 Context 별생성됨 Global Context (context.xml) 모든 Application 에서사용가능 Application Context (application-context.xml) 해당 Application 에서만사용가능

Global Resource Level (+ Resouce Link) Server : ${CATALINA_HOME}/conf/server.xml <Server port="${port.shutdown}" shutdown="argo_instance_shutdown"> <GlobalNamingResources> <Resource name="jdbc/argo" driverclassname="com.mysql.jdbc.driver" url="jdbc:mysql://10.0.1.88:3306/oom" username="root password="uco5fkz3nfgdo/lwb0l+0a==" /> </GlobalNamingResources> </Server> Context(1) : ${CATALINA_HOME}/conf/context.xml <?xml version="1.0" encoding="utf-8" standalone="no"?> <Context> <WatchedResource>WEB-INF/web.xml</WatchedResource> <ResourceLink global="jdbc/argo" name="jdbc/argo" type="javax.sql.datasource"/> </Context> Context(2) : ${CATALINA_HOME}/conf/Catalina/localhost/${CONTEXT_NAME}.xml <?xml version="1.0" encoding="utf-8" standalone="no"?> <Context docbase="/sorc001/argo/application/sample-simpleweb" path="/" reloadable="false"> <ResourceLink global="jdbc/argo" name="jdbc/argo" type="javax.sql.datasource"/> </Context>

Context Level Context(1) : ${CATALINA_HOME}/conf/context.xml <?xml version="1.0" encoding="utf-8" standalone="no"?> <Context> <WatchedResource>WEB-INF/web.xml</WatchedResource> <Resource name="jdbc/argo" driverclassname="com.mysql.jdbc.driver" url="jdbc:mysql://10.0.1.88:3306/oom" username="root password="uco5fkz3nfgdo/lwb0l+0a==" /> </Context> Context(2) : ${CATALINA_HOME}/conf/Catalina/localhost/${CONTEXT_NAME}.xml <?xml version="1.0" encoding="utf-8" standalone="no"?> <Resource name="jdbc/argo" driverclassname="com.mysql.jdbc.driver" url="jdbc:mysql://10.0.1.88:3306/oom" username="root password="uco5fkz3nfgdo/lwb0l+0a==" /> </Context>

Commons DBCP 는 Single Thread 를사용한다. 그래서 Thread 가안전할수있도록전체풀 (pool) 에대해서 lock 을건다. 멀티코어 /CPU 환경에서성능이떨어진다. ( 느리다 ) (not DBCP 2.x) Commons DBCP 는복잡하다. Commons DBCP 는관리해야할클래스가 60 여개, Tomcat 은 8 개의핵심클래스만사용한다. Tomcat JDBC 는 Common DBCP 에서크게변경없이전환이가능하며, 추가적으로 JMX 지원 확장등으로외부에서 Connection Validation, JdbcInterceptor 를통한 DB 처리의독립적인처리 추가기능등다양한기능이추가되어있음 소스코드도 Common DBCP 에비해, 상당히간단하며, 릴리즈도 Tomcat 과동시에되기때문에, 버그수정등유지보수에대한안정성 / 신뢰성을제공함. Commons DBCP vs Tomcat JDBC : http://vigilbose.blogspot.kr/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html http://people.apache.org/~fhanik/tomcat/jdbc-pool.html

Jasper Jasper 란 Tomcat 의 JSP 엔진으로, JSP 를컴파일하기위한컴포넌트 JSPC(JSP to Servlet), JavaC(Servlet to Class) 수행

Jasper 모든 JSP 파일은 org.apache.jasper.servlet.jspservlet 과 Servlet 매핑되어있음. <servlet> <servlet-name>jsp</servlet-name> <servlet-class>org.apache.jasper.servlet.jspservlet</servlet-class> <init-param> <param-name>fork</param-name> <param-value>false</param-value> </init-param> <init-param> <param-name>xpoweredby</param-name> <param-value>false</param-value> </init-param>  <load-on-startup>3</load-on-startup> </servlet>  <servlet-mapping> <servlet-name>jsp</servlet-name> <url-pattern>*.jsp</url-pattern> <url-pattern>*.jspx</url-pattern> </servlet-mapping>

정적파일의처리 Static Contents 에대한성능 Apache HTTP Server 의다양한옵션및기능활용 (Access Control, Contents Caching, Filter 등 ) 부하분산 사용자요청에대한부하분산 WEB + WAS 에대한부하분산 장애대응 Fail Over Fail Back 기타 보안 (Security)

웹서버연동방법 mod_proxy - http://httpd.apache.org/docs/2.4/mod/mod_proxy.html mod_jk - http://tomcat.apache.org/connectors-doc/ mod_cluster - http://mod-cluster.jboss.org/ 연동방법비교 항목특징비고 mod_proxy mod_jk mod_cluster Protocal(http, https, ajp) 에관계없이연동가능 Apache HTTP Server 에모듈로포함되어있어, 별도설치및관리가필요없음 기본적인 Load Balancer 기능제공 Apache HTTP Server 프로젝트에서제공 진보된 Load Balancer 기능제공 진보된장애감지기능제공 Apache Tomcat 프로젝트에서제공 적용사례많음 동적으로 worker(tomcat) 추가가가능 worker 상태에따른 Load Balancing 가능 JBOSS 연동을위한모듈 (Tomcat 도연동가능 ) 적용사례가많지않음

Tomcat 프로젝트에서제공하는웹서버연동모듈 Document : http://tomcat.apache.org/connectors-doc/ Download : http://tomcat.apache.org/download-connectors.cgi mod_jk 기능 worker 감시기능 (CPing/CPong) Connection Pooling Sticky Session Retry (Recovery) Load banance worker Status 관리 적용아키텍처

mod_jk 적용방안 mod_jk 설치및설정 (httpd.conf) worker 정의 (worker.properties) WEB-WAS 요청매핑정의 (uriworkermap.properties) mod_jk 설정 LoadModule jk_module modules/mod_jk.so <IfModule mod_jk.c> JkWorkersFile ${INSTALL_PATH}/conf/extra/workers.properties JkLogFile " ${ENGN_HOME}/bin/rotatelogs ${INSTALL_PATH}/logs/jk.log_%Y%m%d 86400 +540" JkLogLevel error JkLogStampFormat "[%a %b %d %H:%M:%S %Y]" JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkShmFile ${INSTALL_PATH}/logs/jk.shm JkMountFile ${INSTALL_PATH}/conf/extra/uriworkermap.properties </IfModule> <Location /jk-status/> JkMount jkstatus Order deny,allow Deny from all </Location>

worker 정의 worker.list=${server_lb_id},jkstatus worker.template.type=ajp13 worker.template.lbfactor=1 worker.template.socket_timeout=300 worker.template.socket_keepalive=true worker.template.connect_timeout=30000 worker.template.connection_pool_size=128 worker.template.connection_pool_minsize=32 worker.template.connection_pool_timeout=20 ######################################### ## Worker for Tomcat ######################################### worker.argow_8180_hostname.reference=worker.template worker.argow_8180_hostname.host=127.0.0.1 worker.argow_8180_hostname.port=8009 ######################################### ## Load Balancer ######################################### worker.${server_lb_id}.type=lb worker.${server_lb_id}.balance_workers=argow_8180_hostname worker.${server_lb_id}.sticky_session=1 worker.jkstatus.type=status

요청매핑정의 /*.jsp /=${SERVER_LB_ID} /*.do /=${SERVER_LB_ID}

status type 의 Worker 설정시 Status Worker 로동작함 Status Worker 는 worker 의상태모니터링및 mod_jk 의설정정보를관리하는기능제공

Clustering 다수의서버를연결하여, 하나의서버로보이게하는기술 Clustering 목적 : 부하분산, 성능향상, 가용성향상, Fail-over Tomcat Clustering Cluster Load Balance (with Apache HTTP Server) Cluster Membership Session Replication

Clustering Arcihitecture

Cluster Architecture Cluster Manager - Cluster 환경에서의 Session Manager 를설정. Channel - cluster group 의 communication 관리를담당. Membership - Multicast(default) 방식으로 cluster group 의노드추가 / 삭제에대한공지를책임짐. Receiver - Member 가보내오는 Message receive 함. Sender- 외부로나가는 Message 를관리. Interceptor - Message send 할때 interceptor stack 을거침. (Send 전 Message 를변경할수있음 ) Deployer - Application Deploy 를 cluster 단위로가능하게해줌. Valve - Request 가 Servlet Container 출입하는시점에 Track.(Tomcat Valve 와동일 ) ClusterListener - Cluster Group 의 Message send/receive 시의 Track.

Cluster Session Manager 종류 구분설명파일명 DeltaManager All-to-All session replication Default Session Manager in Cluster environment For small cluster org.apache.catalina.sessi on.deltamanager BackupManager Primary-Secondary session replication For large cluster org.apache.catalina.sessi on.backupmanager

Session Replication Load Balancer Failover Session (Primary) Load Balancer

Node Recovery

Session Replication (create) Entry Primary: Tomcat #1 Backup: Tomcat #2 Key: JsessionId Value: Session (Value 는 Primary/Backup Node 만동기화된값을가지고있다. Proxy 는 null) Session Replication (destroy)

Failover Node Failure Tomcat1 Access to Primary node Entry Primary: Tomcat #2 Backup: Server #3 Key: JsessionId Value: Session (Value 는 Primary/Backup Node 만동기화된값을가지고있다. Proxy 는 null)

Access to Backup node Accessto Proxy node

Configuration Tomcat Clustering 을위해서는 conf/server.xml 의 <Engine> 또는 <Host> 에설정한다. <Cluster classname="org.apache.catalina.ha.tcp.simpletcpcluster"/> 위설정은아래와동일하다. <Cluster classname="org.apache.catalina.ha.tcp.simpletcpcluster" channelsendoptions="8"> Session Manager 설정 DeltaManager BackupManager <Manager classname="org.apache.catalina.ha.session.deltamanager expiresessionsonshutdown="false" notifylistenersonreplication="true"/> <Channel classname="org.apache.catalina.tribes.group.groupchannel"> <Membership classname="org.apache.catalina.tribes.membership.mcastservice address="228.0.0.4 port="45564 frequency="500 droptime="3000"/> <Receiver classname="org.apache.catalina.tribes.transport.nio.nioreceiver address="auto port="4000 autobind="100 selectortimeout="5000" maxthreads="6"/> <Sender classname="org.apache.catalina.tribes.transport.replicationtransmitter"> <Transport classname="org.apache.catalina.tribes.transport.nio.pooledparallelsender"/> </Sender> <Interceptor classname="org.apache.catalina.tribes.group.interceptors.tcpfailuredetector"/> <Interceptor classname="org.apache.catalina.tribes.group.interceptors.messagedispatch15interceptor"/> </Channel> <Valve classname="org.apache.catalina.ha.tcp.replicationvalve filter=""/> <Valve classname="org.apache.catalina.ha.session.jvmroutebindervalve"/> mod_jk Failover를썼을경우, sessionid 뒤의 jvmroute 값을바꿔줌. <Deployer classname="org.apache.catalina.ha.deploy.farmwardeployer tempdir="/tmp/war-temp/ deploydir="/tmp/war-deploy/" watchdir="/tmp/war-listen/ watchenabled="false"/> <ClusterListener classname="org.apache.catalina.ha.session.jvmroutesessionidbinderlistener"> <ClusterListener classname="org.apache.catalina.ha.session.clustersessionlistener"> </Cluster> 추가로, Application/WEB-INF/web.xml 에 <distributable/> 을추가해준다. 동일 Cluster Group 으로설정하기위해서는 address 와 port (Multicase 주소 ) 가동일해야함. Cluster Message 를받는 Receiver 설정 Port 는충돌방지를위해자동 bind. port <= 자동설정된 port < port + autobind Http Request 중 data 복제여부를설정. txt, html 등일땐 session 이변경되지않으므로 filter 에설정하면됨.

JVM 전반에대해모니터링을제공하는툴 JVM에포함 ($JAVA_HOME/bin/jvisualvm ) 되어있으며, 공식홈페이지에서도다운로드가능 (http://visualvm.java.net/) 제공기능 Display local and remote Java applications Display application configuration and runtime environment Monitor application threads, application performance and memory consumption Take and display thread dumps Take and browse heap dumps

https://github.com/scouter-project/scouter 2015.7.1 initial commit Apache 2 License Java/Scala/Eclipse-RCP

Browser(User) Tomcat(App.) MYSQL(DB) Heap, TPS ResTime Service Profile agent.java agent.host CPU, MEM, DISK agent.host Client Collector

Active Service & Control Indivisual Request Service Perf. & Profile Top-Stack Frequecy Resource Usage & Service Perf.

Recent User Monitoring Objects TPS XLog Active Service Avg Res. Time CPU Usage Heap Usage

L 사는온라인비즈니스매출증대로시스템증설수요가있었으며, 기존 Unix 및상용 WEB/WAS 로는많은투자비가발생하게되어, OSS 의전환 인터넷쇼핑몰등총 22 개업무를 CentOS/Apache/Tomcat 으로전환함

Tomcat 전환이슈 & 해결 HTTPS 에서생성된세션을 HTTP 로이동시인식하지못하는현상 ( 반대의경우, 즉 HTTP 에서생성된세션은 HTTPS 에서도정상적으로사용가능 ) Tomcat Valve 추가개발 (HTTPS 세션생성시, HTTP 세션생성기능 ) JSP 한글깨짐 web.xml 의 jsp-property-group 내 page-encoding 설정 DB Password 노출 Tomcat 기본적으로 DataSource 설정내 DB Password 가 Plain Text 로노출되어있어이를 암호화하여관리하기위한별도의클래스배치

Apache 전환이슈 & 해결 Tomcat Connector 의 Tomcat VM 다운인식지연 OS (VM) 이다운되어있는경우 Tomcat Connector 가이에대한인식까지지연현상발생 ICA (IP Table Control for Apache) 적용하여해결함 간헐적서비스지연현상발생 이미지서버에 mod_expires 를통한이미지브라우저캐시기능적용 js, css 파일대상으로 mod_deflate 적용을통한압축기능적용 응용전환이슈 & 해결 <url-pattern> 중복오류 태그라이브러리버전호환오류 JSP include tag 사용오류 상이한 contenttype 에의한오류 JSP 내공백이없어발생한오류

L 사쇼핑몰전환사례 성능분석 인터넷쇼핑몰병행운영중에 Jeus 와 Tomcat 의서비스응답시간을 Scouter APM 툴로비교해본 결과큰차이가없음

U2L 전환사례 기존에소규모의비핵심업무에만적용했던 OSS 를중요 / 핵심업무에도적용함으로써대규모 Enterprise 환경에도안정적인운영이가능하다는사례를보여줌 기존오픈소스에서는부족한기능을보완할수있는추가적인솔루션을개발하여제공함으로써 추가적인고객 value 를제공가능하였음 Scale-out 이용이한 WEB/WAS 시스템은서버한대의다운으로인한서비스영향이거의없기때 문에적극적인 OSS 전환전략수립이가능해보임

웹서버연동이슈 (mod_jk Fail-over 메커니즘 ) WEB 서버에 sticky 로등록된 WAS 를재시작하거나소스를배포하는경우세션이나머지 WAS 로 Round Robin 방식으로분배되어불필요하게세션이늘어나는증상이발생 정상적인경우세션을맺은 WAS 로만 Request 가전달됨 WAS Internet L4 WEB WAS WEB WAS WAS 서버장애상황인경우, 나머지 WAS 로 Request 가 Round Robin 으로분배 WAS Internet L4 WEB WAS WEB WAS WAS

mod_jk worker lb(load Banancer) : was 와직접적으로통신하지는않고, 멤버로있는실제 worker 에대한관리를수행 lb(load Banancer) - worker 를초기화 - load-balancing factor 를이용하여부하분산 - sticky 세션처리 - 특정 worker 에장애발생시다른 worker 로 failover - load balancer 와멤버의상태정보를제공.

Tomcat 배포판 ( 엔진 ) 하나설치, ${CATALINA_BASE} 를분리 CATALINA_HOME 공유, CATALINA_BASE 분리 서버 library 공유, shell 재사용가능, 설정분리 장점 : 서버라이브러리및설정파일공유로, 관리포인트가줄어듬 단점 : 서버라이브러리공유로패치및버전업그레이드시전체서버에영향을미침 Tomcat Server1(CATALINA_BASE)) Server2(CATALINA_BASE) bin conf lib logs temp webapps work bin conf lib logs temp webapps work bin conf lib logs temp webapps work Tomcat(CATALINA_HOME=CATALINA_BASE) Tomcat (CATALINA_HOME) bin conf lib logs temp webapps work bin conf lib logs temp webapps work [ 서버 N 개서리방식 ] [ CATALINA_HOME 공유, CATALINA_BASE 분리방식 ]

Tomcat 의경우 JAVA 옵션설정시 JAVA_OPTS, CATALINA_OPTS 모두사용할수있으며, 차이는다음과같음 JAVA_OPTS 기동 / 중지시모두적용 Catalina Java Options 등 ex) ## Catalina Java Options (don't modify them) JAVA_OPTS=" ${JAVA_OPTS} -server" JAVA_OPTS=" ${JAVA_OPTS} -DjvmRoute=${JVM_ROUTE}" JAVA_OPTS=" ${JAVA_OPTS} -Dwas_name=${INST_NAME}" JAVA_OPTS=" ${JAVA_OPTS} -Dport.http=${SERVICE_PORT}" JAVA_OPTS=" ${JAVA_OPTS} -Dport.https=`expr ${SERVICE_PORT} + 363`" JAVA_OPTS=" ${JAVA_OPTS} -Dport.ajp=`expr ${SERVICE_PORT} - 71`" JAVA_OPTS=" ${JAVA_OPTS} -Dport.shutdown=`expr ${SERVICE_PORT} - 75`" CATALINA_OPTS 기동시에만적용됨 JVM Memory Options, Business System Java Options ex) ## JVM Memory Options (tune them) CATALINA_OPTS=" ${CATALINA_OPTS} -Xms2048m -Xmx2048m -XX:MaxPermSize=256m" CATALINA_OPTS=" ${CATALINA_OPTS} -verbose:gc" CATALINA_OPTS=" ${CATALINA_OPTS} -Xloggc:${LOG_HOME}/gc_${INST_NAME}.${DATE}.log" CATALINA_OPTS=" ${CATALINA_OPTS} -XX:+UseParallelGC" CATALINA_OPTS=" ${CATALINA_OPTS} -XX:+UseParallelOldGC" CATALINA_OPTS=" ${CATALINA_OPTS} -XX:-UseAdaptiveSizePolicy" CATALINA_OPTS=" ${CATALINA_OPTS} -XX:+PrintGCDetails" CATALINA_OPTS=" ${CATALINA_OPTS} -XX:+PrintGCDateStamps CATALINA_OPTS=" ${CATALINA_OPTS} -XX:+ExplicitGCInvokesConcurrent" CATALINA_OPTS=" ${CATALINA_OPTS} -XX:+HeapDumpOnOutOfMemoryError" CATALINA_OPTS=" ${CATALINA_OPTS} -XX:HeapDumpPath=${LOG_HOME}/hdump ## Business System Java Options (for your application)

setenv 스크립트활용 Catalina 스크립트내에서 setenv 스크립트실행됨 Tomcat 실행스크립트관계 daemon.sh setenv.sh startup.sh catalina.sh setclasspath.sh shutdown.sh Catalina 스크립트 # ----------------------------------------------------------------------------- # Control Script for the CATALINA Server # # Environment Variable Prerequisites # # Do not set the variables in this script. Instead put them into a script # setenv.sh in CATALINA_BASE/bin to keep your customizations separate. # # ----------------------------------------------------------------------------- if [ -r "$CATALINA_BASE/bin/setenv.sh" ]; then. "$CATALINA_BASE/bin/setenv.sh" elif [ -r "$CATALINA_HOME/bin/setenv.sh" ]; then. "$CATALINA_HOME/bin/setenv.sh" fi

Tomcat 설정파일을 Template 방식으로사용 공통 include shell 예제 ex) Tomcat 설정파일 (Server.xml) ex) ## Catalina Java Options (don't modify them) JAVA_OPTS=" ${JAVA_OPTS} -server" JAVA_OPTS=" ${JAVA_OPTS} -DjvmRoute=${JVM_ROUTE}" JAVA_OPTS=" ${JAVA_OPTS} -Dwas_name=${INST_NAME}" JAVA_OPTS=" ${JAVA_OPTS} -Dport.http=${SERVICE_PORT}" JAVA_OPTS=" ${JAVA_OPTS} -Dport.https=`expr ${SERVICE_PORT} + 363`" JAVA_OPTS=" ${JAVA_OPTS} -Dport.ajp=`expr ${SERVICE_PORT} - 71`" JAVA_OPTS=" ${JAVA_OPTS} -Dport.shutdown=`expr ${SERVICE_PORT} - 75`" <?xml version='1.0' encoding='utf-8'?> <Server port="${port.shutdown}" shutdown="argo_instance_shutdown">... <Service name="catalina"> <Connector port="${port.http}" protocol="http/1.1" server="server" connectiontimeout="20000" URIEncoding="UTF-8" redirectport="${port.https}" /> <Connector port="${port.ajp}" protocol="ajp/1.3" server="server" redirectport="${port.https}" maxpostsize="-1" maxthreads="256" connectiontimeout="60000" URIEncoding="UTF-8" acceptcount="10" enablelookups="false" tcpnodelay="true" /> <Engine name="catalina" defaulthost="localhost">... <Host name="localhost" unpackwars="true" autodeploy="false"> <Valve classname="org.apache.catalina.valves.accesslogvalve" directory="${log.home}" prefix="access_${was_cname}." suffix=".log" pattern="%h %l %u %t "%r" %s %b %D" /> </Host> </Engine> </Service> </Server>

- Tomcat Build - Tomcat 확장하기

Tomcat Major Version 별 Documentation 제공 Tomcat8 : http://tomcat.apache.org/tomcat-8.0-doc/ Tomcat Connectors : mod_jk Documentation http://tomcat.apache.org/connectors-doc/ Tomcat Native - http://tomcat.apache.org/native-doc/ Tomcat Wiki http://wiki.apache.org/tomcat/frontpage Migration Guide http://tomcat.apache.org/migration.html Security Report http://tomcat.apache.org.security.html

Mailing List Guide : http://tomcat.apache.org/lists.html Tomcat 에대한공식 Mailing List 정보제공 Mailing List tomcat-users (users@tomcat.apache.org) subscribe: users-subscribe@tomcat.apache.org Tomcat 사용자를위한 Mailing List tomcat-dev (dev@tomcat.apache.org) subscribe: dev-subscribe@tomcat.apache.org Tomcat의개발, 구현및세부기능에대한논의하기위한 Mailing List tomcat-announce (announce@tomcat.apache.org) subscribe: announce-subscribe@tomcat.apache.org Tomcat 릴리즈정보및취약점정보를 Announce 하기위한 Mailing List taglibs-user (taglibs-user@tomcat.apache.org) subscribe: taglibs-user-subscribe@tomcat.apache.org Apache Taglibs(http://tomcat.apache.org/taglibs/) 사용자에대한 Mailing List

Bug Report Guide - http://tomcat.apache.org/bugreport.html Tomcat Community 는버그데이타베이스로 Bugzilla 를사용 Bugzilla 를통하여, 버그검색및버그등록을통한수정요청을할수있음