Week13

Size: px

Start display at page:

Download "Week13"

서훈 초
5 years ago
Views:

1 Week 13 Social Data Mining 02 Joonhwan Lee human-computer interaction + design lab.

2 Crawling Twitter Data OAuth Crawling Data using OpenAPI Advanced Web Crawling

3 1. Crawling Twitter Data

4 Twitter API API REST API Streaming API consumer_key.

5 Twitter

6 Twitter * {!5

7 Twitter Customer Key and Access Token!6

8 Twitter Customer Key and Access Token { {!6

9 Data Formats for Exchange Twitter, Facebook API JSON. JSON (JavaScript Object Notation).,. ( key:value,.!7

10 Data Formats for Exchange JSON(JavaScript Object Notation) {"name2": 50, "name3": " 3", "name1": true} JSON { } " ": " ", " ": 25, " ": " ", " ": " ", " ": [" ", " "], " ": {"#": 2, " ": " ", " ": " "}, " ": " 7 "!8

11 Data Formats for Exchange JSON: Facebook Example!9

12 Data Formats for Exchange XML: Extensible Markup Language W3C. HTML XML.. JSON.!10

13 Data Formats for Exchange XML: Food Menu!11

14 Using JSON from Python json JSON import json json_data = json.loads(json_string) * json_data python dictionary!12

15 Twitter tweepy OAuth token tweepy api. import tweepy # OAuth setup consumer_key = 'YOUR-CONSUMER-KEY' consumer_secret = 'YOUR-CONSUMER-SECRET' access_token = 'YOUR-ACCESS-TOKEN' access_secret = 'YOUR-ACCESS-SECRET' auth = tweepy.oauthhandler(consumer_key, consumer_secret) auth.set_access_token(access_token, access_secret) api = tweepy.api(auth)!13

16 Twitter api = tweepy.api(auth) my_timeline = api.home_timeline() for tweet in my_timeline: print(tweet.text) >> go home. " " _-;; CDMA " "., ~... # _!.

17 Twitter Streaming APIs API REST API: request. Public Streaming API User Streaming API Site Streaming API!15

18 Twitter Streaming APIs Public Streaming API 1% 400 Global Trends User Streaming API Site Streaming API user stream!16

19 Streaming Tweet Data Streaming API. StreamListener lister. Stream. Stream Twitter API.!17

20 2. OAuth

21 Facebook Login Facebook User C Login Password Login Password Facebook User A Facebook DB Password Login Password Login Login Password Facebook User (Me) Facebook User B Facebook User D

22 OAuth OAuth 3rd party. 3 ID, Access Token. OAuth,,,,,,.!20

23 OAuth Facebook App Access Privilege Request OAuth token Facebook User C Access Privilege Facebook User A Facebook DB Request OAuth token Access Privilege OAuth token Facebook User (Me) OAuth token OAuth token Facebook User B Facebook App Request OAuth token Access Privilege Facebook User D

24 3. Crawling using OpenAPI

25 OpenAPI Twitter. Twitter OpenAPI. OpenAPI OpenAPI.!23

26 OpenAPI API OpenAPI. application. application app-key, -.,,. ( call.)!24

27 OpenAPI data.go.kr ( 3.0)!25

28 OpenAPI data.seoul.go.kr ( )!26

29 OpenAPI data.seoul.go.kr ( )!26

30 OpenAPI data.seoul.go.kr ( )!26

31 OpenAPI 제공 사이트 data.seoul.go.kr (서울시 열린데이터광장)!26

32 4. Advanced Web Crawling 1

33 Dynamic Websites AJAX. no page reload ( : facebook, twitter)!. ( : media daum )!28

34 developer tools inspector crawl ( json )!29

35 Developer Tools Chrome (Safari ): View Developer Developer Tools Network!30

36 Developer Tools Recoding Filter comment!31

37 Developer Tools ( +postid )!32

38 Developer Tools parentid=0&offset=0&limit=3&sort=recommend!33

39 5. Advanced Web Crawling 2

40 Crawling Using Selenium & Webdriver Selenium & Webdriver Selenium: Python libraries for automating web browsers pip install selenium : load url, click link Selenium drive driver. Firefox driver: Chrome driver: chromedriver/downloads!35

41 Using Selenium Sample Code from selenium import webdriver url = "..." driver = webdriver.firefox() driver.get(url) element = driver.find_element_by_xpath("// div[@class='alex_more']") element.click() html = driver.page_source soup = BeautifulSoup(html, "html.parser") ## process soup!36

42 xpath xpath xpath(path) driver.find_element_by_xpath( //h1') <h1>~</h1>...xpath( //div') <div <div <div class="header"><a href="" btn_next")]') <ul class="paging"><li>~</li></ul>, <li> class btn btn_next!37

43 Questions?

Microsoft PowerPoint - web-part03-ch19-node.js기본.pptx

Microsoft PowerPoint - web-part03-ch19-node.js기본.pptx 과목명: 웹프로그래밍응용 교재: 모던웹을 위한 JavaScript Jquery 입문, 한빛미디어 Part3. Ajax Ch19. node.js 기본 2014년 1학기 Professor Seung-Hoon Choi 19 node.js 기본 이 책에서는 서버 구현 시 node.js 를 사용함 자바스크립트로 서버를 개발 다른서버구현기술 ASP.NET, ASP.NET