NoSQL 김형준 (gruter) babokim@gmail.com Revision: 2011.06.19 2011 JCO 11th Conference Session ${track_#}-${session_#} Javacommunity.Org
김형준 babokim@gmail.com(gtalk) 그루터, www.gruter.com www.jaso.co.kr www.cloudata.org www.cloumon.org www.twitter.com/babokim www.facebook.com/babokim 페이스북그룹 : 클라우드컴퓨팅구현기술
Thinking BigData http log ( 수십 ~ 수백 GB/day) HELP!!! Web Page Crawler ( 수천억 ~ 수조 URL) ( 수십만페이지 /day) Mobile phone call request ( 수억 /day) Social Network Data ( 수억 /day) More Money!!! Big Server
Is it Possible???
Thinking Data Characteristic Consistency 모든사용자가반드시동일한데이터를봐야하는가? 페이스북은??? Availability 언제든지데이터를저장 / 조회할수있어야하는가? Twitter overload whale Durability 모든데이터가 100% 안정적으로저장되어야하는가? 1 억개중에 1 ~ 2 개유실되면? Data Size 데이터양이증가해도계속저장할수있는가?
What is NoSQL? Next Generation Databases mostly addressing some of the points being non-relational, distributed, open-source and horizontal scalable. The original intention has been modern web-scale databases. NoSQL Anti RDBMS, NoSQL = Not Only SQL popularized in early 2009.
NoSQL 출현배경 Data Tsunami 40 billions Web page, 55 trillions Web link 281 exa-bytes, 45 GB/person 데이터저장소의확장성에대한요구증가 Scale up 방식이아닌 Scale out 방식요구 대용량데이터처리에불필요한기능 UPDATEs and DELETEs and JOIN ACID Transactions Fixed Schema 대용량처리에필요한기능은지원하지않음 hierarchical data, graphs ACID vs. BASE Atomic, Consistency, Isolation, Durability Basically Available, Soft-state, Eventually consistent
CAP(Brewers Conjecture) RDBMS Availability Consistency 분산환경에서적절한응답시간이내에세가지속성을만족시키는 저장소는구성하기어렵다. Bigtable Cloudata HBase Partition Tolerance Dynamo Cassandra http://lpd.epfl.ch/sgilbert/pubs/brewersconjecture-sigact.pdf
CAP(MySQL-Read) Update Data = A Where Key = k1 ClientA Select Data Where Key = k1 ClientB MySQL - master/slave Master Slave #1 Slave #2 replication replication Availability Partition Tolerance Update Data = A Where Key = k1 ClientA Patitioned Select Data Where Key = k1 ClientB Master Slave #1 Slave #2 replication
Not my words, BUT It s from USA
선택의문제
Deep Dive into NoSQL Deep Dive into NoSQL
NoSQL 특징 단순한데이터모델 Key/value, Document 기반, Simple Column 모델 Schema Free Big Data 지원 다수의저가 x86 서버로구성 데이터파티션및복제 Eventually consistent / BASE (not ACID) Simple API 범용적인용도가아닌제한된용도로사용
NoSQL 솔루션 http://nosql-database.org/ Wide Column Store / Column Families Hbase, Cassandra, Hypertable, Cloudata, Amazon SimpleDB, SciDB, Stratosphere Document Store MongoDB, CouchDB, Terrastore, ThruDB, OrientDB, RavenDB, Citrusleaf, SisoDB Key Value / Tuple Store MEMBASE, Riak, Redis, Chordless, GenieDB, Scalaris, Tokyo Cabinet / Tyrant, Scalien, Berkeley DB, MemcacheDB, Hibari, HamsterDB, Pincaster, RaptorDB Object Databases Db4o, Versant, Objectivity, Starcounter, Perst, ZODB, NEO, PicoLisp, Sterling, Morantex 종류도다양하고 솔루션도많다.
Who using NoSQL? Twitter Cassandra, HBase, Hadoop, Scribe, FlockDB, Redis Facebook Cassandra, HBase, Hadoop, Scribe, Hive Netflix Amazon SimpleDB, Cassandra Digg Cassandra SimpleGeo Cassandra StumbleUpon HBase, OpenTSDB Yahoo! Hadoop, HBase, PNUTS Rackspace Cassandra 많은업체에서사용하는일반화된기술이다. 하나이상을사용한다.
NoSQL 분석시고려사항 Data Model Key/Value, Document, Wide Columnar CAP Consistency, Availability Data Indexing Row only, Field indexing API model Basic API: get, put, delete Advance API: execute, mapreduce Data partitioning DHT, META Data replication 지원 / 미지원, Consistency Membership Changes 쉽다 / 어렵다 Master Model Master/Slave, Active/Standby
NoSQL 대표아키텍처 Bigtable How can we build a distributed db on top of Distributed File System? Shared Disk or Data http://labs.google.com/papers/bigtable.html 2006 Dynamo How can we build a distributed hash table appropriate for the data center? DHT (Distributed Hashing Table) http://portal.acm.org/citation.cfm?id=1294281 2006
Google Bigtable Google s Data Management System Google App Engine, Analytics, Docs, Earth, etc. A sparse, distributed, persistent multidimensional sorted map Indexed by row key, column key, timestamp In-Memory, On-Disk 데이터는 Google File System 에저장 분산파일시스템의한계극복 Real time transaction, Batch processing 모두만족 MapReduce
Bigtable clone project Cloudata Korea, Gruter Java, 여러종류의파일시스템지원 자체 commit log 시스템 HBase Apache Java, Hadoop 기반 Hypertable Zvents, Baidu C++, Hadoop, KFS
Cloudata Distributed Data Storage semi-structured data store(not file system) 데이터저장을위해분산파일시스템사용 실시간 / 배치처리모두지원 Google Bigtable clone Data Model, Architecture, Features Open source http://www.cloudata.org Goal 500 nodes 200 GB/node, Peta bytes
Cloudata 특징 테이블관리 Create, drop, modify table schema 실시간데이터처리 Single row operation(no join, group by, order by) Multi row operation: like, between 배치프로세싱지원 Scanner, Direct Uploader, MapReduce Adapter 확장성 Automatic table split & re-assignment 신뢰성 데이터파일은분산파일시스템 (Hadoop) 에저장 커밋로그를위해자체 Commit Log 용클러스터구성 Failover 서버장애시수십초 ~ 수분이내다른서버로재할당 Utility Web Console, Shell(simple query), Data Verifier
Cloudata(Hbase) Architecture 사용자애플리케이션 분산 / 병렬컴퓨팅플랫폼 (MapReduce) Cloudata Master Cloudata ( 대용량분산데이터저장소 ) TabletServer #1 TabletServer #2 TabletServer #n 논리적 Table 물리적저장소 분산파일시스템 (Hadoop or other)
Cloudata System Components CloudataMaster failover / event ZooKeeper (Lock Service) event Client CTable Scanner/ Uploader rpc failover / event rpc socket TabletServer #1 (Cloudata) Commitlog Server #1 TabletServer #2 (Cloudata) Commitlog Server #2 TabletServer #n (Cloudata) Commitlog Server #n DataNode #1 (DFS) TaskTracker #1 (Map&Reduce) DataNode #2 (DFS) TaskTracker #2 (Map&Reduce) DataNode #n (DFS) TaskTracker #n (Map&Reduce) Local disk (SATA) Local disk (SATA) Local disk (SATA) : physical server(cloudata) : daemon process(cloudata) : daemon process(other platform) : control : data
Cloudata Data Model 분산된서버에배포 Rowkey Column#1 Column#n TabletA-1 TabletA-2 row #1 row #k row #k+1 rk-1 Ckn ck2 ck1 - Sorted by rowkey - Sorted by column v1, t1 v2, t2 v3, t2 v4, t3 v5, t4 vn, tn TabletA-n row #m row #m+1 row #n Row#1 CF1 Col1-1 Col1-2 Col1-3 Col1-N Row.Key CF2 Col2-1 Col2-2 Col2-K CFn ColN-1 ColN-2 ColN-M Column Column Key Value(t1) Value(t2) Value(tn)
Data Model 예제 (1:N) 1:N 관계 1 user: 1+friends 질의 : 특정사용자의모든친구 RDBMS Cloudata T_USER id(pk) T_FRIEND user_id T_USER_FRIEND row info friend name sex friend_id type <user_id> name sex age <user_id>=type age select * from T_USER, T_FRIEND where T_FRIEND.user_id =? and T_USER.id = T_FRIEND.friend_id List friendkeys = get(rowkey==?) for each friendkeys { get(rowkey==eachkey) } Hbase Schema Design Case Studies (http://www.slideshare.net/hmisty/20090713-hbase-schema-design-case-studies)
Data Model 예제 (N:M) N:M relation 1 student - many courses 1 course - many students RDBMS Cloudata T_Student T_Student T_S_C T_Course row info course id(pk) name s_id c_id id(pk) title s_id name sex age c_id:<type > sex age type teacher_id T_Course row info course c_id title teacher_id s_id:<type>
Data Model 예제 (log data) 1:N relation 로그레코드 : time, ip, domain, url 저장된로그는 5 분, 시간, 일, 주단위로분석 RDBMS T_ACCESS_LOG time ip domain url referer T_ACCESS_LOG Cloudata row http user <time><inc_counter> ip domain url referer login_id login_id
Data 분산및 Lookup Tablet 1 Table = n Tablet, 데이터분산단위 100 ~ 200MB/Tablet, 수천 Tablet/Server Lookup path ROOT Table: Meta의위치저장 META Table: User Tablet의위치저장 User Table: 데이터파일정보저장 Data File: rowkey에대한인덱스저장
Data operation put(key, value) CommitLogServer CommitLog TabletServer MemoryTable Minor Compaction get(key) Searcher 분리된 MapFile#1 (HDFS) 분리된 MapFile#2 (HDFS) Merged MapFile (HDFS) MapFile#1 (HDFS) MapFile#2 (HDFS) MapFile #n (HDFS) Split Major Compaction
Failover Master 장애 Data operation 은정상처리 Table Schema Management, Tablet Split 기능만장애 Multi-Master 로장애대처 TabletServer 장애 Master 에의해 Tablet re-assign 수십초 ~ 수분이내복구 ZooKeeper 장애 3/5 개 node 로클러스터구성, 절대장애발생하지않음 Hadoop NameNode 장애 별도의이중화방안필요 Hadoop 전체장애 Cloudata 클러스터도장애
TabletInputFormat Cloudata MapReduce Hadoop TableA TabletA-1 Tablet A-2 Tablet A-3 Tablet A-N TaskTracker Map Map Task Task Map Task TaskTracker Map Map Task Task Map Task TaskTracker Map Map Task Task Map Task Partitioned by key TaskTracker Reduce Task TaskTracker Reduce Task TableB Tablet B-1 Tablet B-2 DBMS or HDFS META Table
HBase Google Bigtable clone Data Model, Architecture, Features Open source http://hbase.apache.org Client HMaster ZooKeeper (Lock Service) HTable HBaseAdmin RegionServer#1 RegionServer#2 RegionServer#n DataNode #1 (DFS) TaskTracker #1 (Map&Reduce) DataNode #2 (DFS) TaskTracker #2 (Map&Reduce) DataNode #n (DFS) TaskTracker #n (Map&Reduce) Local disk (SATA) Local disk (SATA) Local disk (SATA)
Performance Experiment Cloudata HBase HBase(Cache) Random read 495 578 1,623 Random write 1,223 2,864 8,300 Sequential read 498 600 2,109 Sequential write 1,327 2,635 6,553 Scan 40,329 22,795 30,840 Number of 1000-byte values read/written per second
Bigtable Usecase
Bigtable Usecase AppEngine Datastore Multi-tenancy Data can be clustered together only if it is in the same Bigtable instance All logical tables for a tenant must be packed into the same Bigtable instance Mapping One column family per logical table ColumnFamily
Cloudata Usecase Gruter: www.gruter.com 클라우드컴퓨팅아키텍팅및컨설팅 소셜네트워크분석및서비스 www.searcus.com 소셜네트워크분석서비스 HTTP Application Analysis Storage WebServer WebServer (apache) WebServer (apache) (apache) Distributed Distributed Search Distributed Search Server Server (lucene, Search (lucene, thrift) Server (lucene, thrift) thrift) Distributed Indexer Analysis App. File Storage (HDFS) Cache API (memcached) API WebServer API WebServer WebServer (jetty) (jetty) AppServer ( jetty) AppServer AppServer (thrift) (thrift) (thrift) MapReduce LogCollector (scribe) Crawler Data Storage (Cloudata)
Cloudata Usecase www.searcus.com Twitter Data 저장 17 대, 20 억이상 rows, 4TB, 200 ~ 250GB/server
Cloudata Usecase Rowkey USERINFO FOLLWER FOLLOWING 00000002 1827129 00000002 1827130 00000002 1827131 created_at: 1247056652000 description: 테스트 favourites_count: 3 followers_count: 91712 friends_count: 89010 Id: 54882396 listed_count: 11801 Location: 서울 Name: 김형준 000000006030342 000000007253242 000000014743863 000000014914904 000000015045031 000000015260344 000000015260344 000000015889416 000000016126916 000000018747960 000000018784326 000000019325363 TWITTER Rowkey 000000015360238 000000021827129-00001 in_reply_to_status_id:-1 in_reply_to_user_id: -1 serial_no: 1063883048 source: Twitter for Android text: 오픈소스활동이나엔지니어링은꾸준함이중요하다. 4-5년이상꾸준하게할수있는환경을스스로만들어나가는것이관건 000000021827129-00002 in_reply_to_status_id: 78046819107610624 in_reply_to_user_id: 40249032 serial_no: 1063882962 source: web Text: @youngwookim cassandra 버린다는거는어디나오나요??? 000000021827129-00003
Hbase Usecase(nFractals) nfractals: www.nfractals.com Cloud 기반 CDN 아키텍팅및컨설팅 Cloud 기반컨텐츠다운로드서비스구축및운영 Hadoop, Hbase, MongoDB, Hive 등활용 NoSQL 활용 User access log 저장 Hadoop 저장, Hive 분석, Hbase 저장 Log 분석결과를관리자가웹에서즉시조회 과금데이터및트래픽현황조회 URL 랭킹, 지역별접근분포등
Hbase Usecase(nFractals) App Server Hadoop Hive (Map/Reduce) App Sever App Sever Hbase Hbase REST Google Chart DataSource 20,000 logs/sec - 고객별 ( 수백 ) - 도메인별 - 분석항목 - 5 분주기 (aggregation) - 50 대
Facebook Message Service New Message Service combines chat, SMS, email, and Messages into a real-time conversation Data pattern A short set of temporal data that tends to be volatile An ever-growing set of data that rarely gets accessed chat service supports over 300 million users who send over 120 billion messages per month Cassandra's eventual consistency model to be a difficult pattern to reconcile for our new Messages infrastructure. HBase meets our requirements Has a simpler consistency model than Cassandra. Very good scalability and performance for their data patterns. Most feature rich for their requirements: auto load balancing and failover, compression support, multiple shards per server, etc. HDFS, the filesystem used by HBase, supports replication, end-to-end checksums, and automatic rebalancing. Facebook's operational teams have a lot of experience using HDFS because Facebook is a big user of Hadoop and Hadoop uses HDFS as its distributed file system.
Hbase usecase(adobe) http://hstack.org/why-were-using-hbase-part-1 When we started pushing 40 million records, HBase squeaked and cracked. After 20M inserts it failed so bad it wouldn t respond or restart, it mangled the data completely and we had to start over. HBase community turned out to be great, they jumped and helped us, and upgrading to a new HBase version fixed our problems On December 2008, Our HBase cluster would write data but couldn t answer correctly to reads. I was able to make another backup and restore it on a MySQL cluster We decided to switch focus in the beginning of 2009. We were going to provide a generic, real-time, structured data storage and processing system that could handle any data volume.
Dynamo Amazon s High available key/value store Shopping Cart Availability 동일데이터를여러노드에복사 Consistency Eventual consistency 현재가용한쇼핑카트의정보를보여주고, 추가 / 삭제연산이가능하도록하는것이 Dynamo 의목적 Persistence Berkley DB Transactional store, MySQL, In-memory buffer N,R,W 파라미터 Consistency, Performance, Availability, Durability 수준설정 N: 데이터의복제본수 R, W: 읽기또는쓰기연산에서성공해야하는노드수 read-intensive/low updates application: 3, 1, 3 Amazon Apps 의기본값 : 3, 2, 2 관련오픈소스프로젝트 Scalaris, Voldemort, Ringo
Cassandra Facebook s data store Apache open source http://cassandra.apache.org Hybrid Bigtable: Data Model, In-Memory/On-Disk Data processing Dynamo: Consistent hashing, No Meta data Data Model Keyspace: Database ColumnFamily: Table Column-name: column-key SuperColumn: column group Support Language server: java, thrift client: java, c/c++, php, python 등
Cassandra Architecture Server1 (token=a) Server5 (token=v) Server2 (token=d) Storage Service JMX RPC Daemon (Thrift, Avro) Server4 (token=o) Server3 (token=k) 모든서버는동일한기능수행 (P2P) 각서버는특정범위의키영역을서비스 사용자지정 token 또는부하상황에따른 token 지정 Server2: A D, Server3: D K, Server1: V A 데이터분산 Random Partitioner: Hash(Key) 를이용, 랜덤하게분산 OrderPreservingPartitioner: Key 이용, 순서있게분산 데이터복제 Ring 구성에서 Successor 에복제
Consistency Level Basically Eventual Consistency Replication, Read/Write Consistency Consistency, Performance, Availability, Durability 수준설정 R, W: 읽기또는쓰기연산에서성공해야하는노드수 ONE, QURUM, ALL read-intensive/low updates application Replica=3, Read=One, Write=ALL High Availability Replica=3, Read=QURUM, Write=QURUM
Cassandra Data Model Keyspace: Database ColumnFamily: Table Column-name: column-key SuperColumn: column group Index: Rowkey, Column-name, Super column name SuperColumn ColumnFamily: MailIndexTerm rowkey: jindolk apple m1:10 m2:5... mn:21... ipad rowkey: jaso terma m1:10 m2:5... mn:21... termb Column: name=mail 번호, value=weight
Data operation Write 연산 클러스터내임의의노드로 request Partitioner 가해당데이터를서비스하는노드선택 디스크에 commit log 저장 각노드에서는메모리에데이터저장 Minor compaction, Major compaction Read 연산 클러스터내임의의노드로 request Partitioner 가해당데이터를서비스하는노드선택 R( 복제본 ) 개의결과데이터를기다림
Failover, Node 추가 / 제거 Failover 특정노드장애시복제본에의해서비스 R/W 파라미터값에따라가용성수준결정 Master 서버가없기때문에특정노드장애시에도지속적인서비스가능 Node 추가 / 제거 노드제거 기존 Ring 에신규노드추가 추가시복제본서버로부터데이터복제후서비스투입 Ring 구성변경 다른노드가제거대상노드의데이터를모두복제한후제거 노드추가 부하가높은 Key 범위를자동으로할당 해당 Key 범위의데이터를모두복제한후서비스투입
Cassandra Client Problem Cassandra Server Membership Failure Detection, Live Node Detection Connection Pool Abstraction Thrift/AVRO, Version Hector 3 rd Cassandra client library, MIT License Application Server User Application ORMapper API Wrapper Connection Pool Cassandra Thrift Client AutoDiscover
Cassandra 성능 http://www.jaso.co.kr/426
Cassandra Usecase 국내모금융권
Cassandra Usecase http://about.digg.com/blog/looking-future-cassandra http://www.thebuzzmedia.com/digg-v4-troubles-are-symptom-of-abigger-problem/ Diggs n 1 Friends 특정아이템에대해 digg 한친구목록 SELECT digdate, userid FROM Diggs WHERE userid IN (SELECT friendid FROM Friends WHERE userid = me ) AND itemid = 13084479 ORDER BY `digdate` DESC, `id` DESC LIMIT 4; Column Family: FriendsItem Rowkey: userid Super Column: itemid Column: friend U11 의 friends 가 U1, U3 U12 의 friends 가 U3 U11 digg Item1, U11 digg Item2, U12 diggs Item1 U1, Item1, U11 U2, U3, Item1, U11 U1, Item1, U11 Item2, U11 U2, U3, Item1, U11 Item2, U11 U1, Item1, U11 Item2, U11 U2, U3, Item1, [U11, U12] Item2, U11
MongoDB Document-oriented storage JSON-like data schemas Dynamic queries Full index support Replication and fail-over support Master-Slave replication Replica pair(replica-set) Limited Master-Master Auto-sharding for cloud-level scalability MapReduce for complex aggregation Reference Foursquare, sourceforge, New York Times, Open Source http://www.mongodb.org
MongoDB Sharding 서버 1 서버 2 서버 3 서버 4 mongod (configsvr) mongod (configsvr) mongod (shard1) mongod (shard1) mongod (shard2) mongod (shard2) mongos mongos 클라이언트 로드밸런서 mongos mongos mongod (shard3) mongod (shard3) mongod (configsvr) 서버 5 서버 6
MongoDB 성능 http://www.jaso.co.kr/416
MongoDB Usecase www.timeattack.co.kr 제일기획, 그루터 모바일, 위치기반이벤트, 쿠폰서비스 Replica-Set 구성 Avatar node Geo search
MongoDB Usecase DAUM MyAgora http://www.platformday.com/2011/files/mongodb.pdf
NoSQL 비교 항목 Cloudata/HBase Cassandra MongoDB Consistency Strong Eventual Strong Availability X O X Partition-Tolerance O O O Partition policy Key ordered (META) Hashing Key ordered Key ordered (configsvr) Replication O O O Data Model Column Family Column Family Document Index Row key Column key Row key Colmun key All Fields Language Java Java C Persistence Hadoop Local File Local File Client Protocol Java, Thrift, REST Thrift, Avro API (C, Java, php, ) License Apache Apache AGPL Company Gruter Facebook, Datastax 10gen
Cloumon 클라우드컴퓨팅관리 / 모니터링도구 오픈소스버전, 엔터프라이즈버전 www.cloumon.org ZooKeeper, Cassandra, Hadoop, Hbase 등관리
결론 데이터저장을위한많은솔루션이존재 Oracle, MySQL 만있다는생각은버려야함 먼저시스템의데이터속성과요구사항을파악 (CAP, ACID/BASE) 한시스템에여러솔루션을적용 소규모 / 복잡한관계데이터 : RDBMS 대규모실시간처리데이터 : NoSQL 대규모저장용데이터 : Hadoop 등 적절한솔루션선택 반드시운영중발생할수있는이슈에대해검증후도입필요 대부분의 NoSQL 솔루션은베타상태 ( 섣부른선택은독이될수있음 ) 솔루션의프로그램코드수준으로검증필요 NoSQL 솔루션에대한안정성확보 솔루션자체의안정성은검증이필요하며현재의 DBMS 수준의안정성은지원하지않음 반드시안정적인데이터저장방안확보후적용필요 운영및개발경험을가진개발자확보어려움 요구사항에부합되는 NoSQL 선정필요처음부터중요시스템에적용하기보다는시범적용필요 선정된솔루션검증, 기술력내재화저장소의경우직접개발할필요도있음 많은인터넷업체에서개발 / 사용하고있는저장소를공개 NoSQL 의경우다양한오픈소스가발표되는원인이기도함
감사합니다. Q&A 13:55 ~ 14:05 에이콘부스
이저작물은크리에이티브커먼스코리아저작자표시 - 비영리 - 동일조건변경허락 2.0 대한민국라이센스에따라이용하실수있습니다. This work is Licensed under Creative Commons Korea Attribution 2.0 License.