Towards Personal Mobile Personal Assistants mycompanion 2014.10.10 Young Tack Park School of Computing Soongsil University
Smartphone Trends: : Virtual Personal Assistant Virtual Personal Assistant based on smartphone sensors and artificial intelligence Intelligence Sense Service Towards Personal Mobile Personal Assistants 2 / 34
Sensors!!! Smartphone Sensor Sensing Value GPS 위도, 경도 Mobile Contexts Point of Interests, Routes Accelerometer 가속기 X,Y,Z 값 스마트폰과의근접 MIC 이동수단, Stable, Shaking, Distance from phone Proximity 밝기 자기장강도 / 방향 Brightness of environment Bluetooth Stable or Vibrating Light 중력의 X,Y,Z axis 신호강도 Stable, Vibrating or Front/Back Wi-Fi In Wi-fi zone or disconnect Magnetic 블루투스 ID/ 신호강도 Sound 수준 Orientation Being alone / with someone / communicating with others / Environmental noise(crowded or calm), Monologue, Dialogue etc. Towards Personal Mobile Personal Assistants 3 / 34
Predictive Service of Google NOW Google s VPA Just the right information at the just right time Towards Personal Mobile Personal Assistants 4 / 34
Google s Location/Web History Location information is important for predicting users behavior Towards Personal Mobile Personal Assistants 5 / 34
Virtual Personal Assistants Apple Siri Google NOW Cortana InMind Towards Personal Mobile Personal Assistants 6 / 34
Virtual Personal Assistant Timeline Our Research First Year Second Year Third Year Cognitive Assistant That Learns and Organizes (CALO) Apple Acquires Siri Google NOW Apple Acquires Cue Microsoft Cortana 2003 2008 2010.04 2012.10 2013.10 2014.04 Towards Personal Mobile Personal Assistants 7 / 34
Towards Cognitive Assistants Google Now MS Cortana Next Siri Siri Cognitive Assistant Predictive Assistant Voice Assistant Voice Recognition Cognitive Companion Towards Personal Mobile Personal Assistants 8 / 34
A Simple Scenario: Predictive Assistant SpatioTemporal Knowledge Context Driving to work on Monday Morning 2013. JAN. 09 AM09:40 Prediction Going to SNU 80% Turn right on next signal 90% Classic music 80% Sports article 90% While Driving Towards Personal Mobile Personal Assistants 9 / 34
Sensing, Learning, Reasoning & Prediction Data Learning Machine Learning User Model Reasoning Intelligence Predict Sensing Towards Personal Mobile Personal Assistants 10 / 34
Client/Server Architecture Sensing Spatio-Temporal Personalized Predictive Service Social Network & Web User Preference Physical Logical Social Mobile Learner Preference Towards Personal Mobile Personal Assistants 11 / 34
Where are they going? : Location Prediction Goal Predition Probability GPS, Accelerometer, Audio 입력 Walking Jogging Bus Subway Route Prediction Probability Towards Personal Mobile Personal Assistants 12 / 34
Research Areas for Mobile Prediction Activity Recognition Spatio-Temporal User POI(Points of Interests) and Route Learning Spatio-Temporal Probability Model Real-time Prediction based on the Model GPS, Time POI Learning Action Recognition GPS Data Temporal Data Action Data Route Learning Spatio Temporal Model Real time Sensor Data Spatio-Temporal Location Histoty Real Time Prediction Towards Personal Mobile Personal Assistants 13 / 34
Action Logger Towards Personal Mobile Personal Assistants 14 / 34
Sensor Based Action Learner Wi-Fi Staying GPS 가속도 가속도 Walking Jogging Accelerometer Data 오디오 오디오 Bus Audio Data Wi-Fi Wi-Fi Subway Smartphone Sensor Data Action State Towards Personal Mobile Personal Assistants 15 / 34
Spatio-Temporal User Model Spatio temporal over Points of Interests Spatio temporal probability of taking a particular routes going to a POI Clustering Learner POI and Routes Learner Probabilistic Parameter Model P(POI Time, Loc) P(Route Time, Loc) Probabilistic Reasoner Temporal Location Prediction Time, GPS Acc, Audio Spatio- Temporal Location History Towards Personal Mobile Personal Assistants 16 / 34
Mobile Life Logging L i f e Logging P O I Learner GPS Error Correction using GIS Snapping R o u t e Learner Towards Personal Mobile Personal Assistants 17 / 34
Points of Interests Learning 데이터수집 P O I Learner 이동경로학습 Towards Personal Mobile Personal Assistants 18 / 34
Route Learner P L AY DEMO 데이터수집 주요장소학습 R o u t e Learner Towards Personal Mobile Personal Assistants 19 / 34
Overview of Prediction Model Mobile Life Log Work Home Route1 Route2 POIs Routes Temporal Model Abstract HMM Model Work 직장 Home 체육관 집 Gym Mon Mon 0.61 0.03 0.36 0.11 0.83 0.06 Night Route1 Route2 Work 직장 체육관 Gym Ho 집 me Mon Mon 0.33 0.61 0.33 0.03 0.33 0.36 0.33 0.11 0.33 0.83 0.33 0.06 이용도로 Realtime GPS GIS GIS Snapping GPS, Action, Time 집 체육관 Inference Engine 경로 1 경로 2 POI Prediction Probability Route Prediction Application Towards Personal Mobile Personal Assistants 20 / 34 이용도로 걷기월 / 오후 이용도로 걷기월 / 오후
Spatio Temporal User Model Spatio Temporal User Model Dynamic Bayesian Network g k-1 g k POI t k-1 Switching Part t k Route f g k f t k f ts k 목적지 g k ts k-1 ts k Current Street 이동경로현재위치 t k ts k 행위모드 M k Tk-1 Dk-1 Sk-1 M k-1 T k D k Temporal S k M k 도로 I D 요일정보 시간정보 S k D k T k Temporal Towards Personal Mobile Personal Assistants 21 / 34
Dynamic Bayesian Network Representation G : goal T : Trip TS : Trip Segments k-1 k k+1 k+2 Towards Personal Mobile Personal Assistants 22 / 34
Abstract HMM Model 확률파라메터 g k-1 t k-1 Switching Part g k t k Goalto-Goal Transition CPDs t P( gk gk 1, fk off ) 1, gk gk 1 I( gk 1 gk) t swt _ on P( gk gk 1, fk on) I( gk 1) swt _ on Z k-1 D k-1 ts k-1 S k-1 m k-1 f g k f t k f ts k T k D k ts k S k m k Trip-to- Trip Street- to- Street t P( tk tk 1, gk 1, fk off ) 1, tk tk 1 I( tk 1 tk, gk 1) t swt _ on P( tk tk 1, gk 1, fk on) I( tk 1, gk 1) swt _ on ts P( tsk tsk 1, tk, fk off ) 1, tsk tsk 1 0.2, prev( tsk 1, tk) 0.4, ts, ts in t t P( ts 0.4, ( 1,, ) next tsk 1, tk) k tsk tk fk on I( tsk 1 tsk), tsk 1 not in t I( tsk 1) k 1 k 1 k k Emission CPDs Temporal CPDs Trip-to- Ob. action w1, mk action( tk)) P( mk tk) w2, mk action( tk)) Goal-to- Weekday D P( Z g ) k k D I( Z, g ) k k I( g ) k True street-to- Ob. street 1 P( sk tsk) min_ dist( s, ts ) k k Goal-to- Timeband T P( Z g ) k k T I( Z, g ) k k I( g ) k Towards Personal Mobile Personal Assistants 23 / 34
Spatio-Temporal Conditional Probability P L AY DEMO Towards Personal Mobile Personal Assistants 24 / 34
Prediction Engine 1 POI, Route Learner 2 Probabilistic Graph Model 3 CPD Learner Lifelog Data GPS Data Action Data POI 추출모듈 Trip 추출모듈 Street 추출및 street 맵구성모듈 POI Trip Street 맵 Trip 연결성확인모듈 Trip-street 계수모듈 Street 연결계수모듈 Street 간최단거리계산모듈 Abstract HMM 모델확률자료구조 조건부확률학습엔진 Forward Pass 계수모듈 우도최대화파라미터연산모듈 장소 /Trip/street 예측확률통합연산모듈 Backward Pass 계수모듈 우도계산모듈 Realtime Data GPS Data Action Data Street 인식모듈 4 가중치계산모듈 Particle 생성모듈 정규화모듈 리샘플링모듈 Real time Prediction Trip 사후확률계산모듈 주요장소사후확률계산모듈 Goal Prediction Route Prediction Towards Personal Mobile Personal Assistants 25 / 34
Real time Route Prediction Given : Current Context (GPS, Action, Time), Learned Personal Spatio-Temporal Model Identify plausible goal and route based on real time data Use an approximate reasoning such as a particle filtering algorithm Towards Personal Mobile Personal Assistants 26 / 34
Prediction using Approximate Reasoner Spatio-Temporal Model Shopping G o a l School R o u t e Route01 A c t i o n BUS School Particle Filter Algorithm Shopping Route02 Shopping Route01 Church Route02 School Route01 Church Route01 학교 Route02 Gym Route02 School Route03 Gym Route01 Gym 사용자위치및시간정보 Cgurch Towards Personal Mobile Personal Assistants 27 / 34
Prediction using Approximate Reasoner Spatio-Temporal Model Shoppinh School Particle Filter Algorithm Gym 사용자위치및시간정보 Church Towards Personal Mobile Personal Assistants 28 / 34
Particle Filtering Testing P L AY DEMO Towards Personal Mobile Personal Assistants 29 / 34
Goal and Route Prediction Abstract HMM Trip 8 / Goal-1 Trip 12 / Goal-2 Trip 18 / Goal-3 Particle Filter Real Time Prediction Engine Goal Prediction Probability Input sequence Route Prediction Probability Towards Personal Mobile Personal Assistants 30 / 34
Plausible Goal and Route Prediction P L AY DEMO Towards Personal Mobile Personal Assistants 31 / 34
Predictive Assistant Robust probabilistic model to cope with uncertainties Approximate reasoning for realtime prediction Calendat Gmail Spatio-Temporal User 토픽모델 Learner Latent Dirichlet Allocation 모델 시공간토픽모델 Web Services Spatio-Temporal Physical Learner Hierarchical HMM 모델 SNS Blog News EM 기반시공간 HHMM 파라메터학습 Ziny News SNS Big Contents Collection Web Service 인터페이스 Predictive Service Card Handler Predictive Service 사용자인터페이스 Predictive Service Transportation Preference Model Model Blackboard Particle Filter Contextual Reasoner Task Reasoner Reasoner Service Calendar Agents Agent Predictive Agent Predictive Assistant 시공간 Logger Activity Recognition GPS App News Logger Social Magazine 가속도 Call/SMS SNS Big Contents Classification SNS Big Contents Curation Action Learner GPS, 가속도센서데이터수집 Gaussian Mixture Model 기반분류 Decision Tree 기반분류 Towards Personal Mobile Personal Assistants 32 / 34 32
Predictive Assistant vs Google NOW Google NOW Activity Recognition Google NOW Google NOW Prediction Activity Location Recognition History mycompanion mycompanion mycompanion 본과제결과물 Prediction Activity Location History Prediction Towards Personal Mobile Personal Assistants 33 / 34 33
Thank you!!
클러스터링기반주요장소학습 Action Reasoner 장소및경로추론 확률모델학습 실시간추론 적용방법 사용자 GPS 이력을기반으로함 일정시간같은지점에 stay action 이인지된좌표를 POI Candidates 로설정 DBSCAN 을적용하여밀집된 Candidates 를하나의 POI 로인지 DBSCAN(density-based spatial clustering of applications with noise) 공간상에분포한대량의데이터들에대하여다양한크기와모양의클러스터를탐지하는기법 주요장소후보군 주요장소인지 DBSCAN 수행 Towards Personal Mobile Personal Assistants 35 / 34
이동경로학습 Action Reasoner 장소및경로추론 확률모델학습 실시간추론 시간, 장소, Action 로그 이동경로학습 이동경로학습 Towards Personal Mobile Personal Assistants 36 / 34
이동경로클러스터링 Action Reasoner 장소및경로추론 확률모델학습 실시간추론 유사한이동경로추출 5 월 11 일 5 월 5 일 5 월 18 일 이동경로클러스터링 이동경로데이터 Towards Personal Mobile Personal Assistants 37 / 34