Pattern Recognition

SELF-DRIVING CARS AND DEEP LEARNING 아주대학교구형일

Course overview Introduction Self Driving Cars/Machine Learning/Deep Learning Machine Learning Artificial Neural Network (ANN,MLP) Convolution Neural Network (CNN) Recurrent Neural Network (RNN) (Deep) Reinforcement Learning

Course overview with applications with (some) codes

INTRODUCTION

교통사고원인 1. Driver distraction 2. Speeding 3. Drunk driving 4. Reckless driving 5. Rain 6. Running red lights 7. Running stop signs 8. Teenage drivers 9. Night driving 10. Design defects 11. Unsafe lane changes 12. Wrong-way driving 13. Improper turns 14. Tailgating 15. Driving under the influence of drugs 16. Ice 17. Snow 18. Road rage 19. Potholes 20. Drowsy driving 21. Tire blowouts 22. Fog 23. Deadly curves 24. Animal crossings 25. Street racing 26. Others

TED: Sebastian Thrun

SELF-DRIVING CARS

DARPA Grand Challenge II (2006)

DARPA Urban Challenge (2007)

Autonomous-driving is hard The world is complex The world is unpredictable The world is hazardous

Moravec s Paradox The main lesson of 35 years of AI research is that the hard problems are easy and the easy problems are hard. The mental abilities of a four-year-old that we take for granted recognizing a face, lifting a pencil, walking across a room, answering a question in fact solve some of the hardest engineering problems ever conceived... As the new generation of intelligent devices appears, it will be the stock analysts and petrochemical engineers and parole board members who are in danger of being replaced by machines. The gardeners, receptionists, and cooks are secure in their jobs for decades to come. Pinker, Steven (September 4, 2007) [1994], The Language Instinct, Perennial Modern Classics, Harper, ISBN 0-06-133646-7

Moravec s Paradox

Moravec s Paradox

Moravec s Paradox RoboCup 2016: NimbRo vs AUTMan

Moravec s Paradox A Compilation of Robots Falling Down at the DARPA Robotics Challenge

Why? Encoded in the large, highly evolved sensory and motor portions of the human brain is a billion years of experience about the nature of the world and how to survive in it. We are all prodigious Olympians in perceptual and motor areas, so good that we make the difficult look easy. Abstract thought, though, is a new trick, perhaps less than 100 thousand years old. We have not yet mastered it. It is not all that intrinsically difficult; it just seems so when we do it Moravec, Hans (1988), Mind Children, Harvard University Press

How hard is driving?

Deep learning to the rescue DNN BIG DATA GPU The GPU is the workhorse of modern A.I. POPULAR SCIENCE

SELF DRIVING CARS

The basic self-driving loop MAP SENSE CONTROL LOCALIZE PLAN PERCEIVE

Autonomous Driving 자율주행레벨 * Level 0 단순경고 Level 1 단순 ADAS Level 2 퓨전 ADAS Level 3 제한적자율주행 Level 4 완전자율주행 차간거리유지 Adaptive Cruise Control 현대제네시스 ( 15.12) 구글카 ( 개발중 ) 차간거리 / 차선유지 Smart Cruise Control 제한적자율주행 Auto Pilot 완전자율주행 상용화기능 차선이탈경고 Lane Departure Warning System 전방추돌경고 Forward Collision Warning System 차선유지 Lane Keeping Assist 자동긴급제동 Advanced Emergency Braking 차선변경 Lane Change Assist 측 / 후방경고 Blind Spot Warning System 주차보조자동주차원격자동주차 무인발렛주차 법제도 허용개선필요불허 *National Highway Traffic Safety Administration (NHTSA- 미국고속도로교통안전청 ) 기준

GOOGLE S SELF DRIVING CAR

Google self-driving car GPS LiDAR Camera Radar

Visualization of LIDAR data

Disengagements Reports Disengagements: deactivations of the autonomous mode when a failure of the autonomous technology is detected (272 cases) when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle. (13+56 cases)

California Autonomous Testing Disengagements (2015) https://www.wired.com/2017/02/california-dmv-autonomous-car-disengagement/

TESLA S AUTOPILOT

https://www.tesla.com/

Tesla vs Google

Google 과 Tesla 의 자율주행자동차기술차이 Computer Vision LIDAR 사용 ( 높은위치인식능력 ) Camera 사용테슬라의 CEO 앨런머스크는구글의 LIDAR 센서에관해 그렇게비싼센서를사용한자율주행자동차를개발하는것은과하다 (overkill) 라고비판 Car Control 완전자율주행기술목표 2013 년구글은일부직원의출퇴근에자율주행차를타도록했는데차안비디오카메라를모니터링한결과운전자가잠이드는등운전에집중하지않음. 이후완전자율주행기술을목표로개발 Autopilot 기능제공자동차의비행기화 비행기에서돌발상황에만파일럿이개입하듯돌발상황에서운전자의조작이필요

http://www.hankyung.com/news/app/newsview.php?aid=201607018385g

How the Accident happened

Traffic Fatalities Total miles driven in U.S. in 2014: 3,000,000,000,000 (3 million million) Fatalities: 32,675 (1 in 90 million) Tesla Autopilot mile driven since October 2015: 300,000,000 (300 million) Fatalities: 1

UBER

Pittsburgh, your self-driving Uber is arriving now

UBER/nuTonomy

NVIDIA S DRIVERWORKS

NVIDIA

Perception

Visualization

Planning

COURSE OVERVIEW

Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) Localizati on GPS, SLAM Self Driving Perception Pedestrian detection (HOG+SVM) Detection/ Segmentat ion/classif ication Dry/wet road classificati on ADAS Planning/ Control Optimal control End-toend Learning End-toend Learning Tasks Driver state Behavior Prediction/ Driver identificati on Vehicle Diagnosis Smart factory DNN * * Reinforcement * Unsupervised * *

ARTIFICIAL NEURON

ADAS Tasks Self Driving Localizati on Perception Planning/ Control Driver state Vehicle Diagnosis Smart factory Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) DNN Reinforcement GPS, SLAM Pedestrian detection (HOG+SVM) Detection/ Segmentat ion/classif ication Dry/wet road classificati on Optimal control End-toend Learning End-toend Learning Unsupervised

뉴런 : 신경망의기본단위

인공뉴런 (Artificial Neuron) 실제뉴런 뉴런의수학적모델

예시 : 연어와농어의구별 폭 (w) 7.3l + 3.4w = 100 밝기 (l) l w 농어 연어 7.3l + 3.4w 100 7.3l + 3.4w < 100

4 2 2 4 예시 : 연어와농어의구별 l 7.3 l 1.0 0.8 w 3.4 w Σ 0.6 0.4 0.2 연어 / 농어 100

Artificial Neuron w T x g(w T x) Activation function (non-linear)

Multi-layer Perceptron

TYPES OF MACHINE LEARNING Neural Network 기반방법을중심으로

Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) Localizati on GPS, SLAM Self Driving Perception Pedestrian detection (HOG+SVM) Detection/ Segmentat ion/classif ication Dry/wet road classificati on ADAS Planning/ Control Optimal control End-toend Learning End-toend Learning Tasks Driver state Behavior Prediction/ Driver identificati on Vehicle Diagnosis Smart factory DNN * * Reinforcement * Unsupervised * *

Why neural networks? Universal function approximator

Why neural networks? It can learn

Why neural networks? There can be lots of variations (layouts)

Types of Machine Learning Supervised Learning Classification/Regression Semi-supervised Learning/Weakly supervised Learning/ Unsupervised Learning Clustering Feature Learning Generative Model Learning Reinforcement Learning Deep Q-Learning Policy Gradient Learning

Supervised learning workflow

Supervised vs unsupervised Supervised Learning Data: (x, y) x is data, y is label Goal: Learn a function to map x -> y Examples: Classification, regression, object detection, semantic segmentation, image captioning, etc Data: x Unsupervised Learning Just data, no labels! Goal: Learn some structure of the data Examples: Clustering, dimensionality reduction, feature learning, generative models, etc

Unsupervised Learning Generative Model (Generative Adversarial Network)

Unsupervised Learning Generative Model (Generative Adversarial Network)

Unsupervised Learning Dimension Reduction/Feature Learning (Auto-Encoder)

Supervised vs Reinforcement Supervised Learning Data: (x, y) x is data, y is label Goal: Learn a function to map x -> y Examples: Reinforcement Learning reward x i, y i = (, ) Game state Joystick control

Reinforcement Learning Reinforcement learning is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.

Reinforcement Learning

DEEP LEARNING

Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) Localizati on GPS, SLAM Self Driving Perception Pedestrian detection (HOG+SVM) Detection/ Segmentat ion/classif ication Dry/wet road classificati on ADAS Planning/ Control Optimal control End-toend Learning End-toend Learning Tasks Driver state Behavior Prediction/ Driver identificati on Vehicle Diagnosis Smart factory DNN * * Reinforcement * Unsupervised * *

구글트렌드 : 딥러닝

Neural network Back propagation, Nature 1986 장점 일반적인문제에적용할수있는학습법 Biological 시스템과관련이깊음 문제점 Training 이쉽지않음 현실적인문제에잘동작하지않음

Neural network Back propagation, Nature Non-linear SVM 1986 1992 다양한시도들 Flat structure SVM, Boosting, Biological 시스템과거리가생김 특정한문제를해결하는특정한방법 (SIFT, LBP, HOG, GMM-HMM)

Neural network Back propagation, Nature Deep belief network, Science 1986 1992 2006 비지도학습을이용한 pre-training Training 방법향샹 Dropout, RectLinear, Normalization, 컴퓨터구조의발달 GPU Multi-core computer 시스템 빅데이터 +

Neural network Back propagation, Nature Deep belief network, Science Speech 1986 1992 2006 2011

음성인식성능

Neural network Back propagation, Nature Deep belief network, Science Speech Object recognition 1986 1992 2006 2011 2012 Submission Method Error rate Supervision Deep CNN 0.16422 ISI XRCE/INRIA OXFORD_VGG FV: SIFT, LBP, GIST, CSIFT FV: SIFT and color 1M-dim features FV: SIFT and color 270K-dim features 0.26172 0.27058 0.27302

ImageNet Large Scale Visual Recognition Competition (ILSVRC) Steel drum Output: Scale T-shirt Steel drum Drumstick Mud turtle Output: Scale T-shirt Giant panda Drumstick Mud turtle http://www.image-net.org/challenges/lsvrc/

Neural network Back propagation, Nature Deep belief network, Science Speech Object recognition 1986 1992 2006 2011 2012 2013 IMAGENET 2013: 영상인식 RANK Name Error rate Description 1 NYU 0.11197 Deep Learning 2 NUS 0.12535 Deep Learning 3 OXFORD 0.13555 Deep Learning

Neural network Back propagation, Nature Deep belief network, Science Speech Object recognition 1986 1992 2006 2011 2012 2013 2014 IMAGENET 2013: 영상인식 RANK Name Error rate Description 1 Google 0.06656 Deep Learning 2 Oxford 0.07325 Deep Learning 3 MSRA 0.08062 Deep Learning

Neural network Back propagation, Nature Deep belief network, Science Speech Object recognition The game of GO 1986 1992 2006 2011 2012 2013 2014 2016

The AI race is on IBM Watson Achieves Breakthrough In Natural Language Processing Facebook Launches Big Sur Baidu Deep Speech 2 Beats Humans Google Launches TensorFlow Toyota Invests $1B In AI Labs Microsoft & U.Science & Tech, China Beat Humans on IQ

Deep Learning: Representation Learning

The Mammalian Visual Cortex is Hierarchical

Deep Learning: Scalable Machine Learning

딥러닝모델의특징 다층구조 (multi layer) 신경망의구조모사 상위층으로갈수록추상화된정보가학습과정에서자동으로생성 문제해결과정자동화 End-to-end learning 사람의개입을배제하고오직 raw input 과 output 사이에모든과정을데이터에서학습하는방향추구 분산표현 Distributed representation 여러뉴런이협력하여정보저장 / 처리

DRAWBACKS

Current Drawbacks Big data: inefficient at learning from data Supervised data: costly to annotate real-world data Need to manually select network structure Need to hyper-parameter tuning Learning rate Loss function Mini-batch size Number of training iterations Momentum Optimizer selection Defining a good reward function is difficult

Faulty Reward Functions in the Wild

BACKUPS Clustering

AI 역사

AI 역사

I m sorry Dave.

AI 역사

딥블루 딥블루 vs 게리카스파로프, 1997 Deep Blue vs Kasparov 3 1 2 vs 2 1 2 Brute-force search power 6~8 수를내다봄

AI 역사

IBM Watson 슈퍼컴퓨터 질문예시 ) Kathleen kenyon s excavation of this city mentioned in joshua showed the wall had been repaired 17 times WHAT is Jericho This child star got his first onscreen kiss in MY GIRL WHO is Macaulay Culkin

AI 역사 2016

CLUSTERING

Unsupervised Learning Clustering To group the object having same feature Female group male group

Unsupervised Learning Clustering

Clustering Example