오픈소스 NoSQL(MongoDB) 을이용한 Twitter Stream 의저장과실시간공간지식발견 군산대학교컴퓨터정보공학과 kwnam@kunsan.ac.kr 남광우
연구배경및필요성 Twitter 스트림에서의공간정보추출 - 도메인온톨로지와텍스트정보에기반한위치추정에집중 - 공간현상의추출및공유부분은부족 Twitter 스트림스트림에서정보추출 - 자연어기반텍스트정보셋에서의키워드추출 - 시간의변화에따른이슈변화모니터링 - 단일키워드기반의정보제공, 현상에대한경험적추가지식필요 데이터마이닝기술의활용 - 추출된키워드들간의유용한연관관계를발견 - 사용자에게이슈 ( 키워드 ) 에대한이해도를향상시킬수있음 SNS 에서데이터마이닝기법을이용공간상에논의되는다양한이슈들추출이를시맨틱공간지식 (GeoSemantic Knowledge) 으로가공, 활용가능 2
연구내용 Twitter 스트림에서의시공간지식추출프레임워크개발 스트림마이닝을활용한 SNS 데이터수집및이슈추출 대용량 Twitter 스트림의저장과분석프레임워크개발 Twitter Stream 의축적과빠른검색 NoSQL(MongoDB) 를이용한저장 빠르며반복적인데이터접근을지원하기위한데이터 Caching Sliding Window 기반의 Twitter 스트림분석 공간연관규칙과 Clustering 트위터에서의공간지식추출활용방안 특정지역에서의시간에따른이슈변화모니터링 3
MongoDB in NoSQLs - 왜 MongoDB 를선택했는가?
NoSQL NoSQL = No SQL? no no... NoSQL = Not Only SQL!!! Relational Graph Key-Value Data Column Document 5
왜 NoSQL 인가? Relational DBMS ACID Transactions 안전성, 완전성중심 성능과보호의절충 구현의복잡성 Many Insert/Many Update Web, BigData NoSQL Massive Data Complex Data Schema Free One Insert/Many Read 성능과안전 구현의단순성 6
NoSQL for BigData http://blog.kissmetrics.com/twitter-statistics/?wide=1 7
NoSQL Stack Apache Hadoop Google Hadoop MapReduce HBase Hadoop Distributed File System(HDFS) Distributed Programming Model Distributed Databases Distributed File System MapReduce BigTable Google File System(GFS) Cluster 8
NoSQL 의종류 Key-value stores Amazon s Dynamo Cassandra BigTable, BerkeleyDB Document Database MongoDB, CouchDB Column stores Hadoop/HBase, Google BigTable, Cassandra Graph databases FlockDB, Neo4J 9
NoSQL : Document Database CouchDB, MongoDB JSON Twitter 에서 JSON 으로데이터제공 10
MongoDB Document Oriented Database Data is stored in documents, not tables / relations JSON -> BSON : BSON is a binary representation of JSON MongoDB is Implemented in C++ best performance Platforms 32/64 bit Windows Linux, Mac OS-X, FreeBSD, Solaris Language drivers for: Ruby / Ruby-on-Rails Java C# JavaScript C / C++ Erlang Python, Perl others... 11
MongoDB Data and Queries JSON : Example location1 = { name: "10gen HQ, address: "17 West 18th Street 8th Floor, city: "New York, zip: "10011, latlong: [40.0,72.0], tags: [ business, cool place ], } tips: [ {user:"nosh", time:6/26/2010, tip:"stop by for office from 4-6pm"}, {...}, ] 12
MongoDB Data and Queries Creating your indexes db.locations.ensureindex({tags:1}) db.locations.ensureindex({name:1}) db.locations.ensureindex({latlong: 2d }) Finding places: db.locations.find({latlong:{$near:[40,70]}}) With regular expressions: db.locations.find({name: /^typeaheadstring/) By tag: db.locations.find({tags: business }) 13
Twitter 데이터의수집과저장 - Streamming API 와 MongoDB
Twitter Streamming API Twitter Streamming API public streams : follow, track, locations, count, with user streams site streams Twitter4J 15
Twitter Streamming API https://stream.twitter.com/1.1/statuses/filter.json? locations=-122.75,36.8,-121.75,37.8 { "text": "Time for the States to fight back!!! Tenth Amendment Movement: Taking On the Feds http://bit.ly/14t1rv #tcot #teaparty, "created_at": "Tue Nov 17 21:08:39 +0000 2009", "geo": -74.5,40.25, "id": 5806348114, "in_reply_to_screen_name": null, "in_reply_to_status_id": null, "user": { "screen_name": "TPO_News", "created_at": "Fri May 15 04:16:38 +0000 2009", "description": "Child of God - Married - Gun carrying NRA Conservative - Right Winger hard Core Anti Obama (Pro America), Parrothead - www.aboldstepback.com #tcot #nra #iphone", "followers_count": 10470, "friends_count": 11328, "name": "Tom O'Halloran", "profile_background_color": "f2f5f5", "profile_image_url": "http://a3.twimg.com/profile_images/295981637/tpo_balcony_normal.jpg", "protected": false, "statuses_count": 21147, "location": "Las Vegas, Baby!!", "time_zone": "Pacific Time (US & Canada)", "url": "http://www.tpo.net/1dollar", "utc_offset": -28800, } } 16
Twitter Stream 의수집과저장 Tweets from following Media Stream API (MediaCollector Account) Internet SNS data collector Tweet collector Abstraction by Lucene Insert Collected data JSON Tweet raw DataBase (MongoDB) Mentions from anonymous users(selected based on location) Tweet Word Vector Databases(MongoDB Tweet collector: Streaming API 기반트윗수집, 중복성검사, 저장 Abstraction Processing: Full Text Tweet 을 Lucene 을이용하여 Word Vector 화 17
Twitter Stream 의수집과저장 TweetStreamCollector Twitter4J void start( String dbname ) { dbmanager = new MongoDBManager( dbname ); TwitterStream stream = new TwitterStreamFactory().getInstance(); stream.addlistener( this ); FilterQuery query = makefilter(); stream.filter( query ); ensureindex(); } 18
Twitter Stream 의수집과저장 Twitter4J 의 Streamming Listener 구현 public void onstatus(status status) { } JSONObject mediadata = makejsonmediadata( status ); dbmanager.insertdata( MongoDBManager.MEDIADATA_COLLECTION, mediadata.tostring() ); String mediadataid = (String)mediaData.get("MediaDataID"); JSONArray wordarray = (JSONArray)mediaData.get( "TermVector"); JSONObject abstracteddata = makejsonabstracted( mediadataid, status, wordarray ); dbmanager.insertdata( MongoDBManager.ABSRACTED_COLLECTION, abstracteddata.tostring()); 19
Twitter Stream 의수집과저장 Syntactic parser: 문장에서분석을위한어휘소추출 Stopword filter: 의미없는분석결과배제를위한필터링 by LUCENE Tweet raw DataBase (MongoDB) Text analyzer Syntactic parser Tweet word set DataBase (MongoDB) Word Dictionary Stopword filter Spam manager 20
Twitter Stream 의수집과저장 Term Vector 의생성 JSONArray makejsontermvector( String text ) throws Exception { TagMagnitudeVector tmv = tweetanalyzer.analyze( text ); JSONObject jsontag = null; List<TagMagnitude> tml = tmv.gettagmagnitudes(); // 분석데이터를담을 JSONArray 생성 JSONArray jsontermvector = new JSONArray(); for( TagMagnitude tm : tml ) { // 단일분석데이터생성 jsontag = new JSONObject(); jsontag.put("displaytext", tm.getdisplaytext()); jsontag.put("stemmedtext", tm.getstemmedtext()); jsontag.put("magnitude", tm.getmagnitude()); } // 분석데이터삽입 jsontermvector.put( jsontag); } return jsontermvector; 21
수집된데이터 V06 약 3.5 일치분량 16G V07 약 3 일치분량 14G 약시간당 15,000 개 75,000 개 Term 22
지식탐사와 Sliding Window Caching
Twitter 데이터스트림에서의이슈추출 스트리밍데이터처리 : Sliding Window 시간의흐름에따라분석대상단어들의모집단이변하기때문에이를다시계산해야하는문제점을슬라이딩윈도우를도입하여보완 [Celi2011, Lee2011a, Math2010, Sank2009] 24
공간지식 : 연관규칙 & co-location Answers: and Spatial Databases : A Tour 25
공간지식 : 연관규칙 & co-location Spatial Databases : A Tour 26
공간지식 : Spatial Clustering/Flock Density-based Clustering For Real-time Stream Data 27
공간지식 : Outlier Detection Traffic Outlier Detection 28
공간지식 : Outlier Detection 29
Twitter Mining Keywords Trends 30
Twitter Mining 31
Twitter Mining 32
Twitter Mining 33
Problems Massive Data Storage Caching Knowledge Association Rule Clustering 34
Caching for Twitter Streams Stream Caching and Analysis Twitter Stream Sliding Window Caching Twitter Twitter Collector Stream Analysis MongoDB Twitter RawData MongoDB Twitter Abstracted Data 35
Caching for Twitter Streams StreamStorage Twitter Stream Sliding Window Caching New Arrival Tweet Streams sliding window query 36
연관규칙탐사알고리즘 : FP-stream FP-Growth 를데이터스트림에활용할수있도록변형 빈번하게발생하는패턴과트리구조가시간의변화에따라민감하게변화하지않을것이라는가정에근거 Pattern tree 에빈발항목을저장하고 tiled-time windows 에현시점까지의빈발항목을축적하여최근빈발한항목에대한변화를용이하게파악 [Gian2003] 37
연관규칙탐사알고리즘 소셜미디어스트림에서의 fp-tree 알고리즘적용제약사항 FP-stream: 고정된크기의시간을사용하기때문에유동적인빈발항목을탐색하기어려움 시간의누적에따른트리복잡도및메모리사용량증가 FUFP-tree: Minimum count 값이상의빈발규칙이새롭게발생할경우, 해당규칙의유효성을검사하기위하여이전에발생한 transaction 들을모두검색해야함 시간의누적 ( 트랜잭션의누적 ) 에따른신규규칙추출확률감소 FP-tree 알고리즘적용을위한요구사항 변화하는시간에따라추출되는이슈의변화를관측할수있어야함 FP-tree 구현을위해운용가능한메인메모리규모를유지해야함 38
메모리관리방안 패턴노드테이블 Pattern node ID(nid) 를메모리에저장 SC i : B i 에서발생한규칙에대한 support count Hash를통해 FP-tree 의패턴노드테이블 (memory or disk) 에접근 Hash table for Pattern-node Time-sensitive Sliding-window Pattern-node table nid : 0010 B 1 B 2 B 3 B B i-3 B i-2 B i-1 B i transaction Root B i SC i B 1 10 B 2 15 B 5 20 a b c nid 0010 c b d nid : 0010 c d d f 39
메모리관리방안 LRU 구조이용 메모리와 Disk 의 Hybrid 저장구조지원 pattern tree를구성하는항목들에대한 time window 메타정보를관리 특정시간 (threshold value) 이상접근되지않을경우 Disk 로저장 or 삭제 t(b i ) t(b i+1 ) New LRU Structure nid 0010 New rules nid : 0010 B i SC i B 1 10 threshold value B 2 15 B 5 20 nid 0010 Past 40
구현 < 데이터수집기 > < 연관성분석기능의분석옵션설정 > < 분석데이터결과매쉬업 > < 데이터분석기 > 41
구현 // long timeunit, long slidetime, long windowtime, long stoptime, minsupport, number of rules exp1.expdstree(10*minute, 20*MINUTE, 2*HOUR, starttime, stoptime, 100, 1000); [Total Nodes :115196:Created:18379:Deleted:17555 : x 0 [FrequencyList Size : 42923 attrnodelist Size : 42923: Slided Out Items : 5008 streamlinesize : [0:2078][1:1994][2:2064][3:2110][4:2133][5:2160][6:2131][7:2173][8:2129][9:2211][10:2 186][11:2133][12:2205] stream Transaction Sum: 27707 stream Items Total Sum: 158901 [Total Nodes :116292:Created:18799:Deleted:17703 : x 0 [FrequencyList Size : 43653 attrnodelist Size : 43653: Slided Out Items : 4971 streamlinesize : [0:2064][1:2110][2:2133][3:2160][4:2131][5:2173][6:2129][7:2211][8:2186][9:2133][10:2 205][11:2144][12:2277] stream Transaction Sum: 28056 stream Items Total Sum: 160736 42
구현 트위터검색연동 구글검색연동 43