MongoDB Daum Communications
NoSQL
Using Java Java VM, GC Low Scalability Using C Write speed Auto Sharding High Scalability Using Erlang Read/Update MapReduce R/U MR Cassandra Good Very Good MongoDB Good Poor but Scalable Cassandra 153% 112% MongoDB 145% 211%
MongoDB?
MyAgora
60,000K 52,445,415 45,000K 32,445,415 30,000K 24,350,302 18,925,573 15,000K 4,596,409 8,445,415 13,605,034 0...... 2011-01 2011-02 2011-03 2011-04 2011-05 2011-08 2012-01
Needs NoSQL!! Legacy Too many inserts Short period
$mysql->query("alter TABLE test ADD INDEX idx_id(id)"); for($i = 0 ; $i < $limit ; $i++) { $q = "insert into test values($i,'$i')"; $mysql->query($q); } for($i = 0 ; $i < $limit ; $i++) { $q = "select * from test where id = $i"; if ($result = $mysql->query($q)) while ($row = $result->fetch_assoc()) {} } for($i = 0 ; $i < $limit ; $i++) { $q = "update test set value = '$i$i' where id = $i"; $mysql->query($q); } for($i = 0 ; $i < $limit ; $i++) { $q = "delete from test where id = $i"; $mysql->query($q); } $col->ensureindex(array("id"=>1)) for($i = 0 ; $i < $limit ; $i++) { $col->insert(array("id"=>$i,"value"=>"$i")); } for($i = 0 ; $i < $limit ; $i++) { $c = $col->find(array("id"=>$i))->hint(array ("id"=>-1))->limit(1); foreach($c as $a) {} } for($i = 0 ; $i < $limit ; $i++) { $col->update(array("id"=>$i), array('$set'=>array ("value"=>"$i$i"))); } for($i = 0 ; $i < $limit ; $i++) { $col->remove(array("id"=>$i)); } 1,000,000 Data Insert/Select/Update/Delete Performance Insert Select Update Delete MySQL 249.39 sec 354.82 sec 316.82 sec 310.37 sec MongoDB 59.37 sec 293.99 sec 153.39 sec 123.34 sec
My
MyAgora
Agora MYSQL Disk Disk MyAgora 2011-02-07 205,874 2011-02-08 220,909 2011-02-09 184,568 Agent Mongo DB Mongo DB Mongo DB
Backup & Recovery
fsync and lock Data dumping (mongodump) Backup Shutdown backup Data export (mongoexport)
04:00AM Latest Mongo DB Daily Backup Server Mongo DB Mongo DB Weekly Monthly
All Master Backup file Mongorestore Master Slave to Master Server Slave Server Sync Slaves Server
Practical Tips
Data Modeling Index Performance
Schemaless RDB Data Join Application Mashup
Index
B-tree 1 Query - 1 Index Wrong/No index hint() High disk IO Memory Size Too big index Compound index Need Sharding
findone(query) find(query).hint(index).limit(1) Log Scale 10000 findone find.hint 5.493 3.443 3.494 1000 3.970 3.610 100 10 0.032 0.030 0.025 0.030 0.015 1 5 / 6.64G 4 / 4.84G 3 / 3.74G 2 / 2.1G 1 / 1.38G
Performance
Connection Singleton Pattern Connection Pooling slaveok()
[ERROR] 2010-11-17 14:19:24 org.apache.catalina.core.standardwrappervalve.invoke (StandardWrapperValve.java:253)-Servlet.service() for servlet dispatcher threw exception com.mongodb.mongointernalexception: can't find a master at com.mongodb.dbtcpconnector.checkmaster(dbtcpconnector.java:327) at com.mongodb.mongo.<init>(mongo.java:215) at com.mongodb.mongo.<init>(mongo.java:200) Caused by: com.mongodb.mongoexception$network: can't call something at com.mongodb.dbtcpconnector.call(dbtcpconnector.java:210) at com.mongodb.dbtcpconnector.call(dbtcpconnector.java:208) at com.mongodb.dbtcpconnector.call(dbtcpconnector.java:208) at com.mongodb.dbapilayer$mycollection. find(dbapilayer.java:256) at com.mongodb.dbcollection.findone(dbcollection.java:467) at com.mongodb.dbcollection.findone(dbcollection.java:456)
private static MongoDBConnection instance = null; private static Mongo m = null; private MongoDBConnection() { try { List<ServerAddress> mongodbhosts = getmongodbhosts(); if (mongodbhosts.size() == 1) { m = new Mongo(mongodbHosts.get(0)); } else { m = new Mongo(mongodbHosts); } m.slaveok(); } catch (UnknownHostException e) { e.printstacktrace(); } } private static List<ServerAddress> getmongodbhosts() throws UnknownHostException { String hostspop = mongodbprop.getstring(key_mongodb_hosts); String [] hostsarray = StringUtils.split(hostsPop, ","); String port = getmongodbport(); List<ServerAddress> hosts = new LinkedList<ServerAddress>(); for (String host : hostsarray) { hosts.add(new ServerAddress(host.concat(":").concat(port))); } return hosts; }
Cursor Timeout Exception batchsize count() vs cursor.size() count() cursor.size() 0.537 sec 0.158 sec
$cursor->limit(30)->batchsize(7) // if we iterate through 28 items, the next call to getnext() will contact the // database and request a batch of 2 documents $cursor =$relationshipcollection->find()->limit()->hint(); while($cursor->hasnext()){ $relationship = $cursor->getnext(); } $batchsize = 1000; $cursor = $relationshipscollection->find()->hint(); $cursor = $cursor->batchsize($batchsize); while($cursor->hasnext()){ $relationship = $cursor->getnext(); }
Update Object update Atomic update Reindexing update() update($set) 0.387 sec 0.032 sec
public void updatemy(my my, String intro) { BasicDBObject query = new BasicDBObject(); query.put(key_daum_id, my.daumid); query.put(key_cp_code, my.cpcode); my.intro = intro; DBObject dbo = MongoDBUtil.toDB(my); DBCollection col = MongoDBUtil.getProfileCollection(); col.update(query, dbo); } public void updatemy(my my, String intro) { BasicDBObject query = new BasicDBObject(); query.put(key_daum_id, my.daumid); query.put(key_cp_code, my.cpcode); DBCollection col = MongoDBUtil.getProfileCollection(); DBObject dbo = MongoDBUtil.toDB(my); col.update(query, new BasicDBObject("$set", new BasicDBObject("intro", intro))); }
Data file 64mb -> 128mb -> 256mb -> 512mb -> 1.0gb -> 2.0gb db.repairdatabase() 1. Data 30GB 2. Data => Data 30GB 3. Data Insert => If(inserted Data > 30GB) then Data
DB Profiling Slow Query system.profile Only latest operations
Level Setting 0 off 1 log slow operations 2 log all operations > use admin switched to db admin > db.setprofilinglevel(2); { "was" : 0, "slowms" : 50, "ok" : 1 } > db.setprofilinglevel(1, 200); { "was" : 2, "slowms" : 50, "ok" : 1 } > db.system.profile.find({millis : { $gt : 5 }}) {"ts" : "Thu Jan 29 2009 15:21:27 GMT-0500 (EST)", "info" : "query test.foo ntoreturn:0 exception bytes: 53", "millis" : 88}
Full table scan Wrong Index mongodb.conf count() logpath=/daum/logs/mongodb.log logappend=true replset=mongoset1 notablescan=true > db.articles.find({"classname":""}); error: { "$err" : "table scans not allowed:test.articles", "code" : 10111 }
OR Mapper (JB)Son <-> Object Morphia @preload @postload @notsaved
Tools
Rock Mongo
Rock Mongo
mongostat
mongostat
Compatible Light Transaction Log type data Area / MongoDB More Data Modeling Sharding
Thanks rockfactory@daumcorp.com