NoSQL

Size: px

Start display at page:

Download "NoSQL"

정화 육
6 years ago
Views:

1 MongoDB Daum Communications

2 NoSQL

Read/Update MapReduce R/U MR Cassandra Good Very Good

3 Using Java Java VM, GC Low Scalability Using C Write speed Auto Sharding High Scalability Using Erlang Read/Update MapReduce R/U MR Cassandra Good Very Good MongoDB Good Poor but Scalable Cassandra 153% 112% MongoDB 145% 211%

6 MongoDB?

7 MyAgora

8 60,000K 52,445,415 45,000K 32,445,415 30,000K 24,350,302 18,925,573 15,000K 4,596,409 8,445,415 13,605,

9 Needs NoSQL!! Legacy Too many inserts Short period

10 $mysql->query("alter TABLE test ADD INDEX idx_id(id)"); for($i = 0 ; $i < $limit ; $i++) { $q = "insert into test values($i,'$i')"; $mysql->query($q); } for($i = 0 ; $i < $limit ; $i++) { $q = "select * from test where id = $i"; if ($result = $mysql->query($q)) while ($row = $result->fetch_assoc()) {} } for($i = 0 ; $i < $limit ; $i++) { $q = "update test set value = '$i$i' where id = $i"; $mysql->query($q); } for($i = 0 ; $i < $limit ; $i++) { $q = "delete from test where id = $i"; $mysql->query($q); } $col->ensureindex(array("id"=>1)) for($i = 0 ; $i < $limit ; $i++) { $col->insert(array("id"=>$i,"value"=>"$i")); } for($i = 0 ; $i < $limit ; $i++) { $c = $col->find(array("id"=>$i))->hint(array ("id"=>-1))->limit(1); foreach($c as $a) {} } for($i = 0 ; $i < $limit ; $i++) { $col->update(array("id"=>$i), array('$set'=>array ("value"=>"$i$i"))); } for($i = 0 ; $i < $limit ; $i++) { $col->remove(array("id"=>$i)); } 1,000,000 Data Insert/Select/Update/Delete Performance Insert Select Update Delete MySQL sec sec sec sec MongoDB sec sec sec sec

11 My

12 MyAgora

13 Agora MYSQL Disk Disk MyAgora , , ,568 Agent Mongo DB Mongo DB Mongo DB

14 Backup & Recovery

15 fsync and lock Data dumping (mongodump) Backup Shutdown backup Data export (mongoexport)

16 04:00AM Latest Mongo DB Daily Backup Server Mongo DB Mongo DB Weekly Monthly

17 All Master Backup file Mongorestore Master Slave to Master Server Slave Server Sync Slaves Server

18 Practical Tips

19 Data Modeling Index Performance

20 Schemaless RDB Data Join Application Mashup

21 Index

22 B-tree 1 Query - 1 Index Wrong/No index hint() High disk IO Memory Size Too big index Compound index Need Sharding

23 findone(query) find(query).hint(index).limit(1) Log Scale findone find.hint / 6.64G 4 / 4.84G 3 / 3.74G 2 / 2.1G 1 / 1.38G

24 Performance

25 Connection Singleton Pattern Connection Pooling slaveok()

26 [ERROR] :19:24 org.apache.catalina.core.standardwrappervalve.invoke (StandardWrapperValve.java:253)-Servlet.service() for servlet dispatcher threw exception com.mongodb.mongointernalexception: can't find a master at com.mongodb.dbtcpconnector.checkmaster(dbtcpconnector.java:327) at com.mongodb.mongo.<init>(mongo.java:215) at com.mongodb.mongo.<init>(mongo.java:200) Caused by: com.mongodb.mongoexception$network: can't call something at com.mongodb.dbtcpconnector.call(dbtcpconnector.java:210) at com.mongodb.dbtcpconnector.call(dbtcpconnector.java:208) at com.mongodb.dbtcpconnector.call(dbtcpconnector.java:208) at com.mongodb.dbapilayer$mycollection. find(dbapilayer.java:256) at com.mongodb.dbcollection.findone(dbcollection.java:467) at com.mongodb.dbcollection.findone(dbcollection.java:456)

27 private static MongoDBConnection instance = null; private static Mongo m = null; private MongoDBConnection() { try { List<ServerAddress> mongodbhosts = getmongodbhosts(); if (mongodbhosts.size() == 1) { m = new Mongo(mongodbHosts.get(0)); } else { m = new Mongo(mongodbHosts); } m.slaveok(); } catch (UnknownHostException e) { e.printstacktrace(); } } private static List<ServerAddress> getmongodbhosts() throws UnknownHostException { String hostspop = mongodbprop.getstring(key_mongodb_hosts); String [] hostsarray = StringUtils.split(hostsPop, ","); String port = getmongodbport(); List<ServerAddress> hosts = new LinkedList<ServerAddress>(); for (String host : hostsarray) { hosts.add(new ServerAddress(host.concat(":").concat(port))); } return hosts; }

28 Cursor Timeout Exception batchsize count() vs cursor.size() count() cursor.size() sec sec

29 $cursor->limit(30)->batchsize(7) // if we iterate through 28 items, the next call to getnext() will contact the // database and request a batch of 2 documents $cursor =$relationshipcollection->find()->limit()->hint(); while($cursor->hasnext()){ $relationship = $cursor->getnext(); } $batchsize = 1000; $cursor = $relationshipscollection->find()->hint(); $cursor = $cursor->batchsize($batchsize); while($cursor->hasnext()){ $relationship = $cursor->getnext(); }

30 Update Object update Atomic update Reindexing update() update($set) sec sec

31 public void updatemy(my my, String intro) { BasicDBObject query = new BasicDBObject(); query.put(key_daum_id, my.daumid); query.put(key_cp_code, my.cpcode); my.intro = intro; DBObject dbo = MongoDBUtil.toDB(my); DBCollection col = MongoDBUtil.getProfileCollection(); col.update(query, dbo); } public void updatemy(my my, String intro) { BasicDBObject query = new BasicDBObject(); query.put(key_daum_id, my.daumid); query.put(key_cp_code, my.cpcode); DBCollection col = MongoDBUtil.getProfileCollection(); DBObject dbo = MongoDBUtil.toDB(my); col.update(query, new BasicDBObject("$set", new BasicDBObject("intro", intro))); }

32 Data file 64mb -> 128mb -> 256mb -> 512mb -> 1.0gb -> 2.0gb db.repairdatabase() 1. Data 30GB 2. Data => Data 30GB 3. Data Insert => If(inserted Data > 30GB) then Data

33 DB Profiling Slow Query system.profile Only latest operations

34 Level Setting 0 off 1 log slow operations 2 log all operations > use admin switched to db admin > db.setprofilinglevel(2); { "was" : 0, "slowms" : 50, "ok" : 1 } > db.setprofilinglevel(1, 200); { "was" : 2, "slowms" : 50, "ok" : 1 } > db.system.profile.find({millis : { $gt : 5 }}) {"ts" : "Thu Jan :21:27 GMT-0500 (EST)", "info" : "query test.foo ntoreturn:0 exception bytes: 53", "millis" : 88}

35 Full table scan Wrong Index mongodb.conf count() logpath=/daum/logs/mongodb.log logappend=true replset=mongoset1 notablescan=true > db.articles.find({"classname":""}); error: { "$err" : "table scans not allowed:test.articles", "code" : }

36 OR Mapper (JB)Son <-> @notsaved

37 Tools

38 Rock Mongo

39 Rock Mongo

40 mongostat

41 mongostat

42 Compatible Light Transaction Log type data Area / MongoDB More Data Modeling Sharding

43 Thanks

PowerPoint 프레젠테이션

PowerPoint 프레젠테이션 Reasons for Poor Performance Programs 60% Design 20% System 2.5% Database 17.5% Source: ORACLE Performance Tuning 1 SMS TOOL DBA Monitoring TOOL Administration TOOL Performance Insight Backup SQL TUNING