Dynamic Warehousing 구현을위한리얼타임데이터통합 배상균 SWG, IP&S 팀 2007 IBM Corporation
IOD Framework for Dynamic Warehousing Complete, Integrated Information Infrastructure Banking Retail Telco Public Sector Etc. Etc. Expertise & Accelerators Business Strategy and Planning Services Industry Models and Pre-Built Solution Assets Integrated IBM Partner Components IBM Master Data Management Server Define & Synch Manage Transact IBM DB2 Warehouse Reliable Real-Time Delivery Party, Product, Account 데이터통합서비스 IBM Information Server Manage Analyze Understand Cleanse Transform Deliver Unified Metadata Management Parallel Processing Rich Connectivity 2 2007 IBM Corporation
Business value 증대를위한데이터통합의요건 급변하는업무환경과고객요구에적절히대응하기위해 IT 조직은양질의데이터를, 적절한시점에, 필요한시스템으로전달할수있어야합니다. Business Value 데이터분석을통한의미있는정보생산에소요되는시간과 Business value 는반비례 Business event 이상적인데이터통합의요건 Value lost Data ready for analysis Information delivered Action taken Right Data Right Time Right System Action time or Action distance Time Value of Data Source: Richard D. Hackathorn, Bolder Technology, Inc. Time 3 2007 IBM Corporation
데이터통합을위한 IBM 의플랫폼 양질의데이터를, 적절한시점에, 필요한시스템으로전달할수있기위해서는다양한기능과서비스를유기적으로통합해서제공해야합니다. Source Systems Source Connectivity Assessment and Validation Standardization & Alignment Cleansing & Harmonization Transformation & Movement Target Connectivity Target Other ERP Implementation Methodology CRM Understand Cleanse Transform Deliver MDM SCM RDBMS Discover, model, and govern information structure and content Standardize, merge, and correct information Combine and restructure information for new uses Platform Services Synchronize, virtualize and move information for in-line delivery ERP/SCM Parallel Processing Services Connectivity Services Metadata Services Administration Services Deployment Services Legacy Data BI/DW Web Services XML, EDI IBM Information Server 4 2007 IBM Corporation
데이터통합기술발전경향 3 가지방향 Real-time Integrated Platform Service Oriented Source: Gartner, 2007 5 2007 IBM Corporation
Real-time 데이터통합요구사항의증가 기업내의다양한시스템 /DB 간데이터통합은갈수록실시간성격을많이요구받고있습니다. Drivers Requirements Benefits Action-based ROI Inhibitors Gathering Transformation Aggregation Analysis Data cleansing Movement Cost Source: Gartner, 2007 6 2007 IBM Corporation
Real-time 데이터통합의확대필요성 데이터통합인프라는배치성격과실시간성격의적절한조화를통하여기업의 Real-time 요건을만족시켜야합니다. 다양한시스템간 Data Latency 의최소화 Real-time 성격주기적 Batch Real-time 성격 주기적 Batch 기업에서운영하는다양한시스템 Real-time 비중의증대 7 2007 IBM Corporation
CDC : 시스템간 Data Latency 최소화를위한핵심기술 Front-end 시스템에서발생하는트랜잭션데이터를다양한분석시스템으로실시간으로변환및전달하기위해서 Real-time CDC 를통한변경데이터추출이기술적으로중요합니다. CDC : Changed Data Capture 다양한 CDC 방법 ODBC/JDBC Adapters SQL Query ( 추출할레코드구분을위해 Timestamp 활용 ) Database Triggers 변환데이터로깅을위한유저테이블 File Comparison ( 최종 CDC 시점의전체데이터와비교 ) DB 로그에서직접추출 8 2007 IBM Corporation
CDC : Real-time 데이터통합의기반기술 DB2 DB2 log CDC 와 ETL 기술의결합 Oracle Oracle Redo log Real-Time CDC DB2 ETL Feeds Data Warehouse I5/DB2 OS/400 Journal Corporate ETL Server 9 2007 IBM Corporation
IBM Information Server 의데이터통합패턴 Federation Consolidation Analytical & Reporting Tools Web Applications Federation Consolidation Analytical & Reporting Tools Web Applications Product Performance Real-time Inventory Level Region 1 Product Performance Region 2 Product Performance Headquarters Primary Data Center Replication Replication Stores Backup Data Center 이기종플랫폼지원을통한솔루션보완과제 Change Data Capture Database Capture and Change PublishData Capture Consolidation 을위한핵심기술 Replication ebusiness Application Information Server Message Queue Target app /DB 10 2007 IBM Corporation
DataMirror 합병을통한 CDC, 복제솔루션강화 IBM 은현재 Mainframe 의다양한 DB 및 DB2 에대한 CDC, 복제솔루션을가지고있습니다. (WebSphere Replication Server, Event publisher...) DataMirror 합병을통해이기종플랫폼및 DBMS 를지원하는방향으로 CDC 및복제솔루션을보완할계획입니다. DataMirror, IBM company 2007 년 9 월에 IBM 에합병 2008 년 1Q 에조직통합완료 A leading provider of heterogeneous real time Changed Data Capture (CDC) and replication solutions 2,300 customers worldwide 15,000 licenses deployed Over $100M and 12 years of R&D invested Headquarters in Markham, Ontario International Offices in North America, South America, Europe, and Asia Publicly traded (TSX: DMC) 11 2007 IBM Corporation
DataMirror 솔루션과 Information Server DataMirror Real-Time Changed Data Capture (CDC) Heterogeneous Replication and Synchronization High-Availability / Disaster Recovery (HA/DR) IBM Information Server Understand Cleanse Transform Deliver Discover, model, and govern information structure and content Standardize, merge, and correct information Combine and restructure information for new uses Synchronize, virtualize and move information for in-line delivery Platform Services Parallel Processing Services Connectivity Services Metadata Services Administration Services Deployment Services 12 2007 IBM Corporation
DataMirror 고객사 13 2007 IBM Corporation
DataMirror 제품군 DataMirror Integration Suite 14 2007 IBM Corporation
DataMirror 의핵심제품 실시간데이터통합을위한 DB 로그기반의 CDC - LiveAudit, Event Server 등의추가옵션 Embeddable Java RDB iseries 솔루션 HA 구축을위한데이터복제 Oracle 솔루션 HA 구축을위한데이터복제 15 2007 IBM Corporation
DataMirror 제품의활용 Dynamic Warehousing & Business Intelligence and Reporting Near-real time 성격의데이터를통한분석및리포팅 생산시스템과 e-business 어플리케이션간의데이터통합 e-business 를위한실시간데이터제공 Real-time Event Detection 특정한이벤트에대한실시간모니터링 HA 구성및장애복구 효율적인방식의데이터복제를통한 HA 구현 16 2007 IBM Corporation
DataMirror 제품의활용 Use case Real-Time Integration Need up-to-date information delivered to a data warehouse 운영시스템 DW/Mart Need up-to-date information delivered for ecommerce applications Downtown Store Midtown Store Inventory Uptown Store 17 2007 IBM Corporation
DataMirror 제품의활용 Use case Replication Need to report on operational systems but cannot impact production system must synchronize data across 1 or more databases Less expensive DBs 소스시스템부하최소화 Need to maintain business continuity during application migrations, consolidations, upgrades 18 2007 IBM Corporation
DataMirror 제품 개요 DataMirror 는다양한 DBMS 에서발생하는데이터변경내용을실시간으로 Capture 하여, 해당데이터가필요한시스템으로데이터를변환및복제해주는데이터통합도구입니다. 1. DBMS 로그에서데이터 Capture Monitoring and Configuration 3. 변경데이터적용 Database TCP/IP Message Queue DB2, Oracle, SQL Server, etc Database Logs Source Engine Target Engine Flat files 2. 변경데이터전송 운영시스템부하최소화를위한 Push 방식의데이터복제 19 2007 IBM Corporation
DataMirror 제품 CDC 기능 DataMirror 는변경데이터추출을위해 DBMS 로그직접읽는방식을활용합니다. 로그기반의데이터추출은트리거나 DBMS API 를이용하는데이터추출방식이나 Timestamp 를이용한배치기반의데이터추출에비해소스시스템부하를최소화하고, 성능을극대화하는장점을가지고있습니다. 운용시스템의성능에최소한의영향 DBMS와연동 ( 트리거, API 등 ) 하지않는로그방식의데이터추출기법 로그 (LOG) 중심의데이터복제기술을적용하여운영시스템의성능에대한영향을최소화 변화가발생한트랜젹션 ( 데이터 ) 을목표시스템으로전송함으로지연시간을최소화 운영시스템의구조에영향이없음 트랜잭션발생시간을위한 Timestamp 필드의추가가필요없음 변경데이터추출을위해사용중인어플리케이션또는테이블스키마의변경이필요없음 운용중인 IT환경에영향이없음 Peer-to-peer 아키텍처로추가적인하드웨어가필요하지않음 단지변화된데이터의복제기술을적용하므로네트워크부담을최소화 운영시스템의구조에영향이없음 트랜잭션발생시간을위한 Timestamp 필드의추가가필요없음 20 2007 IBM Corporation
DataMirror 제품 다양한플랫폼 /DBMS 지원 DataMirror 는동일 DBMS 의복제뿐만아니라이기종 DBMS 나플랫폼을지원합니다. DATABASES Source & Target DATABASES Target Only MESSAGE QUEUE Support OPERATING SYSTEMS HARDWARE PLATFORMS NETWORK PROTOCOLS DB2 / UDB Teradata JMS OS/400 IBM System i TCP/IP Oracle Netezza MQ Series OS/390 IBM System z Sybase ASE TIBCO AIX IBM System p MS SQL Server WebMethods HP-UX HP-9000 PointBase BEA Solaris Intel MS Windows HP Alpha Linux Sun Itanium 21 2007 IBM Corporation
DataMirror 제품 Audit 기능 소스에서 update, delete 가발생된경우복제된타겟 DB 에서데이터를파악할수가없지만, LiveAudit 기능을활용하면소스 DB 에발생한트랜잭션을정확히파악할수있습니다. 작동방식 Real-Time Database Operational Conversions 아키텍처 Monitoring and Configuration Audit Database Insert Insert DB Logs Update Insert TCP/IP Delete Insert Source Engine Target Engine 22 2007 IBM Corporation
DataMirror 제품 Event Server 소스 DB 에발생한각종트랜잭션에 XML 형태의데이터로변환한이후 Message 기반의솔루션에전달할수있습니다. 이를통해실시간으로발생하는이벤트를감지해서업무시스템을보다고객중심서비스로확장시켜나갈수있습니다. Production System Message Queue EDW Rules-based routing Message Queue Application Process Person 23 2007 IBM Corporation
DataMirror 제품 데이터변환기능 DataMirror 는변경데이터를타겟에그대로적용하는데이터복제뿐만아니라타겟에적용시다양한데이터변환도가능합니다. 데이터변환 1010 0110 Row 필터링 Column 필터링 컬럼값변환 Summary 기능 데이터병합 24 2007 IBM Corporation
DataMirror 제품 Zero programming 기존프로그램과 H/W 구성의변경없이관리도구를통한설정만으로데이터복제를구현합니다. 복제 Wizard 제공 데이터변환설정 25 2007 IBM Corporation
DataMirror 제품 Customizing 기능 User Exit 기능을이용해서타겟데이터적용시점에다양한변환기능을구현할수있습니다. 데이터의일반변환뿐만아니라외부어플리케이션과의연계나다수타겟테이블에데이터적용등의 customizing 이가능합니다. 구현방법 C 프로그램 Stored procedure 26 2007 IBM Corporation
DataMirror 제품 모니터링, 관리툴 자바기반의 GUI 제공 환경설정, 관리, 모니터링에활용 통합스크린에데이터통합프로세스를관리 자동매핑및 drag & drop 방식의데이터변환 Event 로그, 경고, 통계리포팅제공 Easy to use Data volume, operation 별실시간조회 27 2007 IBM Corporation
DataMirror 제품 복제의신뢰성 소스 DB 와타겟 DB 에각각 PUSH 엔진과 APPLY 엔진이직접통신을통해데이터를통합하고메타데이터를관리함으로써복제의신뢰성을보장합니다. 1. Capture 2. Transfer 3a. Apply Source Journal SEQ DATA 1 2 3 4 5..... Transfer 엔진 Apply 엔진 Source 테이블 변경데이터추출도중발생할수있는여러장애상황을고려하여메타데이터를관리함으로데이터의정합성을보장 3b. 확인 메타데이터 Target 테이블 2-phase commit 28 2007 IBM Corporation
구축사례 미국의통신사 변경데이터의 ESB (Enterprise Service Bus) integration Billing 시스템인 AMDOCS 의 Real-time event 발생감지를위한변경데이터전송 데이터유형에따라필요한시스템으로 routing Oracle Redo Log Transaction 1 Transaction 2 Transaction 3 Customer Address XML Document Transaction 1 Transaction 8 Queue 1 Other Application Transaction 4 XML Document Transaction 5 Transaction 6 Transaction 7 Transaction 8 Transaction 9 Transaction 10 Billing Transaction 1 Transaction 2 Transaction 3 Transaction 4 XML Document Transaction 1 Transaction 7 Queue 2 Other Application Customer, Address, Memo, Service, Billing etc Transaction 10 Queue 3 db size = 11 terabytes 40 million trans/day 1GB redo log generated per minute Seibel CRM 29 2007 IBM Corporation
구축사례 미국의 Cable 제공사 14 개지역 Billing 시스템의변경데이터가실시간으로마케팅 DW 로적재 트랜잭션유형및변경시점의 timestamp 가핸드코딩없이 ETL 서버로전송 I5/DB2 OS/400 Journal Regional Billing Application I5/DB2 OS/400 Journal Real-Time CDC ETL Feeds Data Warehouse Regional Billing Application Corporate ETL Server Customer Value Creation Marketing System I5/DB2 OS/400 Journal Real-time CDC 와 ETL 솔루션의결합 30 2007 IBM Corporation
구축사례 교보자동차보험 HA 구성및복제를통한장애대처및 Workload balancing 구현 운영시스템 Active 초당 300~400 건의트렌잭션이지연없이실시간복제 백업시스템 Active HACMP 클러스터시스템 Standby ELS 시스템 장애대처 운영시스템에장애발생시1차로클러스터시스템으로전환 클러스터시스템마저장애발생시 Backup 시스템으로전환 Workload Balancing Backup 시스템은항상 active로운영 후선업무조회및분석시스템인 ELS에최신데이터제공 31 2007 IBM Corporation
로드맵 기존고객에대한보호뿐만아니라 IBM IOD 솔루션과의통합을위한로드맵을가지고있습니다. Product Mission Protect Customer Investment Integrate offerings Extend/Optimize solutions Extend Leadership Leverage capture agents Unified UI for management 2009 Integrate offerings Release DM v6.5 with global translations Tighter integration between DM TS, Rep/EP & DataStage 2008 2007 Expand Capabilities Release IBM branded DataMirror v.6.2 technology systems/standards First phase product integration (DM TS with DataStage) Release DB2 Rep/EP v9.5 & Classic Rep/EP v9.5 32 2007 IBM Corporation
로드맵 2007 IBM IOD conference 공식발표자료 33 2007 IBM Corporation
로드맵 ETL 과통합 ETL 툴인 DataStage 와 2008 년까지완벽한연동기능이구현될예정입니다. 1. Staging DB 통한연동 2. MQ 를통한연동 DataMirror 2 staging area 3 DS/QS job DataMirror 2 MQ 3 DS/QS job 1 database 4 database 현재도활용가능 1 2 database 4 database 3. 파일을통한연동 4. 제품내에통합 DataMirror 2 3 File File File DS/QS job DataMirror 3 DS/QS job DS Custom Operator 1 4 2 1 4 database database 2 database database 34 2007 IBM Corporation
로드맵 Information Server 의진화 IBM Information Server Information Services Director Understand Cleanse Transform Deliver Discover, model, and govern information quality and structure Information Analyzer Business Glossary Standardize, merge, and correct information QualityStage Metadata Server IBM Metadata Workbench Transform and enrich information DataStage Virtualize, synchronize and move information Federation Server Replication Server Data Event Publisher Transformation Server icluster ireflect Parallel Processing Rich Connectivity to Applications, Data, and Content 35 2007 IBM Corporation
로드맵 Dynamic Warehousing IBM Master Data Management IBM Industry Models IBM DB2 Warehouse Party Product Account IBM Information Server Legacy Database.... Legacy Database Legacy Database Legacy Database.... Legacy Database Legacy Database IBM MDM 의변경데이터와레거시시스템과의데이터동기화 운영시스템의변경분데이터가실시간으로 DB2 Warehouse 로통합 Information Server 를통해데이터가정제되고변환되어적재 36 2007 IBM Corporation
IBM IOD 구현을위한기반솔루션 IBM OmniFind Analytics Edition IBM Information Server IBM FileNet BPM WebSphere BPM IBM Information Server IBM Rational Data Architect IBM DB2 Warehouse IBM industry data models IBM WebSphere Customer Center Product Center SOA Infrastructure Dynamic Warehousing Information Integration Replication SOA Master Data Management ecommerce IBM Master Data Management Collaborative MDM Customer Customer / Shipping Industry SOA Business Processes Operational MDM Graphic TBD Product Location Analytical MDM Supplier Account High Availability Event Management Data Quality Management Data Lifecycle Mgmt Failure 37 2007 IBM Corporation
완벽한 Real-time 데이터통합솔루션 Real-time CDC 솔루션인 DataMirror 와 IBM IOD 솔루션의결합은기업에서필요한데이터를 right-time 에통합할수있는완벽한솔루션을제공합니다. Loading into ODS in real-time Shrinking Batch windows Operational BI Event Detection Audit & Internal controls Event Driven SOA Synchronous ebusiness Up-to-Date Web Portals Workload balancing 엔터프라이즈아키텍처지원 데이터통합의 Value 증대 38 2007 IBM Corporation
Thank You 39 2007 IBM Corporation