김형준 http://www.jaso.co.kr http://www.seenal.com babokim@gmail.com 이저작물은크리에이티브커먼즈코리아저작자표시 - 비영리 - 변경금지 2.0 대한민국라이센스에따라이용하실수있습니다.
Agenda NoSQL 개요 NoSQL 특징 NoSQL 솔루션 HBase Architecture HBase Data Model HBase Index/Data File HBase Failover HBase Usecase
NoSQL Differ from classic relational database management systems It does not use SQL as its query language It may not give full ACID guarantees It has a distributed, fault-tolerant architecture NoSQL Anti RDBMS, NoSQL = Not Only SQL popularized in early 2009.
NoSQL 출현배경 Data Tsunami 40 billions Web page, 55 trillions Web link(2009 년 ) 281 exa-bytes, 45 GB/person(2009 년 ) 데이터저장소의확장성에대한요구증가 Scale up 방식이아닌 Scale out 방식요구 기존솔루션은대용량데이터또는비정형데이터에불필요한기능존재 UPDATEs and DELETEs and JOIN ACID Transactions Fixed Schema 대용량처리에필요한기능은지원하지않음 hierarchical data, graphs 글로벌인터넷서비스회사의기술공유 구글, 아마존, 야후, 페이스북등 ACID vs. BASE Atomic, Consistency, Isolation, Durability 전통적인기업데이터속성 Basically Available, Soft-state, Eventually consistent 인터넷기반데이터속성
CAP Conjection Consistency RDBMS Availability 분산환경에서적절한응답시간이내에세가지속성을모두만족시키는 저장소는구성하기어렵다. Bigtable Cloudata HBase Partition Tolerance Dynamo Cassandra http://lpd.epfl.ch/sgilbert/pubs/brewersconjecture-sigact.pdf
MySQL CAP(Read operation) Update Data = A Where Key = k1 ClientA Select Data Where Key = k1 ClientB Master Slave #1 Slave #2 replication replication - Availability - Partition Tolerance Update Data = A Where Key = k1 ClientA Patitioned Select Data Where Key = k1 ClientB Master Slave #1 Slave #2 replication
NoSQL 특징 단순한데이터모델 Key/value Document 기반 단순한 Table/Column 모델 다수의저가 x86 서버로구성 대부분분산아키텍처 단위컴포넌트는장애에취약 확장성, 고가용성지원 Automatic partition or shading Data Replication Automatic Failover and Recovery 관계형데이터베이스에비해약한데이터정합성 Eventual Consistency 범용적인용도가아닌제한된용도로사용
NoSQL 고려사항 Data Model Key/Value, Document, Wide Columnar Storage Model In-memory, persistent, Hybrid Consistency Model Strong, Eventual Data partitioning 미지원, DHT, META Membership Changes 지원 / 미지원, 무정지지원, Data reallocation 여부 Read/write performance Supported Index Row key only, Secondary Index, Full text index Failure handling(or Availability) mechanism Hot-Standby, Cold-Standby Data replication Support Client HTTP(REST), 특정언어만지원, 다양한언어지원 License Commercial, Apache, GPL
Data Systems and Timeframes Offline Processing Near-Real-Time Processing Blocking request/response Hadoop Data warehousing OLAP Message queues (JMS, RabbitMQ, ) CEP(Esper) OLTP: MySQL, Oracle, NoSQL: HBase, Cassandra, 1 hour+ 1 seconds to 1 hour Less than 1 seconds
NoSQL Solution key value cache memcached, repcached, coherence, infinispan, extreme scale, jboss cache, velocity, terracota, redis(?) key value store keyspace, flare, schema free, RAMCloud clustered key value store dynamo, voldemort, Dynomite, SubRecord, Dovetaildb ordered key value store tokyo tyrant, lightcloud, NMDB, luxio, memcachedb, actord tuple store gigaspaces, coord, apache river object database ZopeDB, db4o, Shoal document store CouchDB, MongoDB, Jackrabbit, XML Databases, ThruDB, CloudKit, Perservere, Riak Basho, Scalaris wide columnar store BigTable, Cloudata, Hbase, Cassandra, Hypertable, KAI, Qbase, KDI
Who use NoSQL? Twitter Cassandra, HBase, Hadoop, Scribe, FlockDB, Redis Facebook Cassandra, HBase, Hadoop, Scribe, Hive Netflix Amazon SimpleDB, Cassandra Digg Cassandra SimpleGeo Cassandra StumbleUpon HBase, OpenTSDB Yahoo! Hadoop, HBase, PNUTS Rackspace Cassandra Gruter Hadoop, Cloudata, MongoDB, HBase, Hive, Flume
NoSQL 대표아키텍처 Bigtable How can we build a distributed db on top of Distributed File System? Shared Disk or Data http://labs.google.com/papers/bigtable.html 2006 Dynamo How can we build a distributed hash table appropriate for the data center? DHT (Distributed Hashing Table) http://portal.acm.org/citation.cfm?id=1294281 2006
Google Bigtable Google s Data Management System Google App Engine, Analytics, Docs, Earth, etc. A sparse, distributed, persistent multidimensional sorted map Indexed by row key, column key, timestamp LSM-Tree Log Structured Merge Tree In-Memory, On-Disk 병행사용 데이터는 Google File System 에저장 분산파일시스템의한계극복 Real time transaction, Batch processing 모두만족 MapReduce 플랫폼과쉽게연동 Bigtable clone project Cloudata Korea, Gruter, Hadoop 기반, http://www.cloudata.org HBase Apache, Hadoop 기반, http://hbase.apache.org Hypertable Zvents, C++, Hadoop, KFS
HBase Google BigTable 의기능및설계를기반한오픈소스 http://hbase.apache.org Apache License Powerset 소속개발자가처음개발시작 현재는 Cloudera, Facebook 등에서커밋 뛰어난확장성과안정적인 (?) 데이터저장 수백대규모확장가능 Automatic table split & re-assignment Data Storage 로 Hadoop 을사용 Cluster coordinator 로 ZooKeeper(zookeeper.apache.org) 사용 High write throughput, General read throughput 0.x ~ 수 ms write, 수 ms read 테이블관리 Create, drop, modify table schema 실시간데이터처리 Single row operation(no join, group by, order by) Multi row operation: scan 배치프로세싱지원 Input/OutputFormat, Bulking loading Failover 서버장애시수초 ~ 수분이내다른서버로재할당
HBase/Hadoop 기반소프트웨어스택 사용자애플리케이션 분산 / 병렬컴퓨팅플랫폼 (MapReduce) HMaster HBase ( 대용량분산데이터저장소 ) RegionServer #1 RegionServer #2 RegionServer #n 논리적 Table 물리적저장소 분산파일시스템 (Hadoop)
HBase System Components HMaster failover / event ZooKeeper ZooKeeper (Lock (Lock ZooKeeper Service) Service) (Lock Service) event Client HTable HBase Admin rpc failover / event rpc RegionServer #1 RegionServer #2 RegionServer #n DataNode #1 (HDFS) DataNode #2 (HDFS) DataNode #n (HDFS) Local disk (SATA) TaskTracker #1 (Map&Reduce) Local disk (SATA) TaskTracker #2 (Map&Reduce) Local disk (SATA) TaskTracker #n (Map&Reduce) : physical server : daemon process(hbase) : daemon process(other platform) : control : data
HBase Data Model 분산된서버 (RegionServer) 에배포 TableA Rowkey ColumnFamily#1 ColumnFamily#n Region-1 Region-2 row #1 row #k row #k+1 rk-1 Ckn ck2 ck1 - Sorted by rowkey - Sorted by column v1, t1 v2, t2 v3, t2 v4, t3 v5, t4 vn, tn Region-n row #m row #m+1 row #n Row#1 CF1 Col1-1 Col1-2 Col1-3 Row.Key CF2 Col2-1 Col2-2 Col2-K CFn ColN-1 ColN-2 ColN-M Column Column Key Value(t1) Value(t2) Col1-N Value(tn)
Data 분산및 Lookup Region 1 Table = n Region, 데이터분산단위 100 ~ 200MB/Region, 수천 Region/Server Lookup path ROOT Table: Meta 의위치저장 META Table: User Tablet 의위치저장 User Table: 데이터파일정보저장 Data File: rowkey 에대한인덱스저장 ZooKeeper Root Table M.T1.1000:M1 M.T1.2000:M2 Max:mn Meta Index 에대한 Index Meta Table m1 m2 mn T1.100:U1 T1.200:U2 T1.1000:UN T1.1100:U1 T1.1200:U2 T1.2000:UN xx xx n User Region 에대한 Index User defined Table U1 U2 10 20 100 110 120 200 Data file 의 block 에대한 Index(maxkey, file-offset) 64KB scan HFile(Physical file,sorted by rowkey, column-name)
HBase Index 구조 64KB 64KB 64KB 데이터 로우키 컬럼키 값 타임스탬프 rk1 ck1 value1 T1 rk1 ck2 value2 T2 rk2 ck1 value3 T1 rk3 ck1 value4 T1 rk3 ck2 value5 T1 rk3 ck3 value6 T1 rk4 ck1 value7 T2 인덱스 인덱스 오프셋 rk2.ck1 0 rk3.ck2 64 rk4.ck1 128
Column Oriented Row Oriented Column Oriented <...s....-..babokimbabok im@unitel.co.krs eoul...=... jindolkjindolk@n aver.compusan...... v......s...-..lee lee@gmail.comsuw on... babokim jindolk lee babokim@unitel.co.kr jindolk@naver.com lee@gmail.com seoul pusan suwon id 컬럼파일 mail 컬럼파일 address 컬럼파일 MySQL(ibdata1 파일 )
Data Operaiton put(key, value) CommitLog (HDFS) RegionServer Memory Table Minor Compaction get(key) Searcher 분리된 HFile#1 (HDFS) 분리된 HFile#2 (HDFS) Merged MapFile (HDFS) HFile#1 (HDFS) HFile#2 (HDFS) HFile #n (HDFS) Split Major Compaction LSM-Tree: Log Structured Merge Tree
HBase & MapReduce Hadoop MapReduce Platform TableA Region-1 Region-2 Region-3 Region-N TableInputFormat TaskTracker Map Map Task Task Map Task TaskTracker Map Map Task Task Map Task TaskTracker Map Map Task Task Map Task Partitioned by key TaskTracker Reduce Task TaskTracker Reduce Task TableOutputFormat TableB Region-1 Region-2 DBMS or HDFS META Table
Failover Master 장애 Data operation은정상처리 Table Schema Management, Region Split 기능만장애 Multi-Master로장애대처 RegionServer 장애 Master에의해 Region re-assign 수십초 ~ 수분이내복구 ZooKeeper 장애 3/5 개 node 로클러스터구성, 절대장애발생하지않음 Hadoop NameNode 장애 별도의이중화방안필요 Hadoop 전체장애 HBase 클러스터장애
Client API Native Java Client/API get(get get), put(put put), delete(delete delete) getscanner(scan scan) Non-Java Clients REST server Avro server Thrift server TableInputFormat/TableOutputFormat for MapReduce Hbase as MapReduce source and/or target Hbase Shell Jruby shell adding get, put, scan and admin calls
Client Code Configuration conf = new Configuration(); conf.set(hconstants.zookeeper_quorum, "local01"); conf.set("hbase.zookeeper.property.clientport", "2181"); HBaseAdmin admin = new HBaseAdmin(conf); HTableDescriptor desc = new HTableDescriptor(tableName); desc.addfamily(new HColumnDescriptor(cfName)); admin.createtable(desc); HTable table = new HTable(conf, tablename); Put put = new Put("rk1".getBytes()); put.add(cfname.getbytes(), age".getbytes(), 30".getBytes()); table.put(put); Get get = new Get( rk1".getbytes()); get.addfamily(cfname.getbytes()); Result result = table.get(get); for(keyvalue eachvalue: result.list()) { System.out.println(new String(eachValue.getValue())); }
Bigtable Usecase Google News Persionalization
HBase Usecase Facebook Social plugin Transactional 수집 실시간분석 process over 20 billion events per day (200,000 events per second) with a lag of less than 30 seconds. 실시간 Feedback 배치분석 Analytic
HBase Usecase Facebook Social plugin 실시간 Map/Reduce 개념 데이터수집기 (scribe) 데이터수집기 (scribe) 데이터수집기 (scribe) Write, sync Hadoop /category1/collect_1.dat /category1/collect_2.dat /category1/collect_3.dat ptail Driver Checkpoint Handler Aggregation Store Storage Thrift Server HBase Key1: value Key2: value Key3: value 클라이언트
Cloudata Usecase www.seenal.com 소셜네트워크모니터링및분석서비스 트위터, 블로그원본데이터저장및분석용으로활용 HTTP Application Analysis Storage WebServer WebServer (apache) WebServer (apache) (apache) Distributed Distributed Search Distributed Search Server Server (lucene, Search (lucene, thrift) Server (lucene, thrift) thrift) Distributed Indexer Analysis App. File Storage (HDFS) Cache API (memcached) API WebServer API WebServer WebServer (jetty) (jetty) ( jetty) AppServer AppServer AppServer (thrift) (thrift) (thrift) MapReduce LogCollector (flume, scribe) Crawler Data Storage (Cloudata)
결론 데이터저장을위한많은솔루션이존재 Oracle, MySQL 만있다는생각은버려야함 먼저시스템의데이터속성과요구사항을파악 (CAP, ACID/BASE) 한시스템에여러솔루션을적용 -> 한시스템에여러속성을가지고있는데이터존재 소규모 / 복잡한관계데이터 : RDBMS 대규모실시간처리데이터 : NoSQL 대규모저장용데이터 : Hadoop 등 빠른데이터조회 : 캐쉬적절한솔루션선택 반드시운영중발생할수있는이슈에대해검증후도입필요 대부분의 NoSQL 솔루션은베타상태 ( 섣부른선택은독이될수있음 ) 솔루션의프로그램코드수준으로검증필요 NoSQL 솔루션에대한안정성확보 솔루션자체의안정성은검증이필요하며현재의 DBMS 수준의안정성은지원하지않음 반드시안정적인데이터저장방안확보후적용필요 운영및개발경험을가진개발자확보어려움 요구사항에부합되는 NoSQL 선정필요처음부터중요시스템에적용하기보다는시범적용필요 선정된솔루션검증, 기술력내재화저장소의경우직접개발할필요도있음 많은인터넷업체에서개발 / 사용하고있는저장소를공개 NoSQL 의경우다양한오픈소스가발표되는원인이기도함
감사합니다.