SELF-DRIVING CARS AND DEEP LEARNING 아주대학교구형일
Course overview (keywords) Introduction Self Driving Cars/Machine Learning/Deep Learning Machine Learning Artificial Neural Network (ANN,MLP) Convolution Neural Network (CNN) Recurrent Neural Network (RNN) (Deep) Reinforcement Learning
Course overview with applications with (some) codes
INTRODUCTION
교통사고원인 1. Driver distraction 2. Speeding 3. Drunk driving 4. Reckless driving 5. Rain 6. Running red lights 7. Running stop signs 8. Teenage drivers 9. Night driving 10. Design defects 11. Unsafe lane changes 12. Wrong-way driving 13. Improper turns 14. Tailgating 15. Driving under the influence of drugs 16. Ice 17. Snow 18. Road rage 19. Potholes 20. Drowsy driving 21. Tire blowouts 22. Fog 23. Deadly curves 24. Animal crossings 25. Street racing 26. Others
TED: Sebastian Thrun
Traditional resource waste problem Car utilization rate Energy utilization rate Driving (2.6%) Looking for parking (0.8%) Sitting in congestion (0.5%) Parked (96%) 86% of fuel never reaches the wheels Engine losses Move a person/thing (4~5%) Inertia, Aerodynamics Rolling resistance, Aux. power, Transmission losses Idling Road utilization rate Idle state (90%) Occupied by car (10%)
SELF-DRIVING CARS
DARPA Grand Challenge II (2006)
DARPA Urban Challenge (2007)
Autonomous-driving is hard The world is complex The world is unpredictable The world is hazardous
Moravec s Paradox The main lesson of 35 years of AI research is that the hard problems are easy and the easy problems are hard. The mental abilities of a four-year-old that we take for granted recognizing a face, lifting a pencil, walking across a room, answering a question in fact solve some of the hardest engineering problems ever conceived... As the new generation of intelligent devices appears, it will be the stock analysts and petrochemical engineers and parole board members who are in danger of being replaced by machines. The gardeners, receptionists, and cooks are secure in their jobs for decades to come. Pinker, Steven (September 4, 2007) [1994], The Language Instinct, Perennial Modern Classics, Harper, ISBN 0-06-133646-7
Moravec s Paradox
Moravec s Paradox
Moravec s Paradox RoboCup 2016: NimbRo vs AUTMan
Moravec s Paradox A Compilation of Robots Falling Down at the DARPA Robotics Challenge
Why? Encoded in the large, highly evolved sensory and motor portions of the human brain is a billion years of experience about the nature of the world and how to survive in it. We are all prodigious Olympians in perceptual and motor areas, so good that we make the difficult look easy. Abstract thought, though, is a new trick, perhaps less than 100 thousand years old. We have not yet mastered it. It is not all that intrinsically difficult; it just seems so when we do it Moravec, Hans (1988), Mind Children, Harvard University Press
How hard is driving?
Deep learning to the rescue DNN BIG DATA GPU The GPU is the workhorse of modern A.I. POPULAR SCIENCE
Hard problems for programmers Talk, Read, Walk, Drive, Atari, Easy problems for humans Hard problems for humans Easy problems for programmers Modelbased things (e.g., Physics simulation)
Hard problems for programmers Talk, Read, Walk, Drive, Atari, Deep Learning to the Rescue Easy problems for humans Hard problems for humans Easy problems for programmers Modelbased things (e.g., Physics simulation)
자율주행소개
The basic self-driving loop MAP SENSE CONTROL LOCALIZE PLAN PERCEIVE
Autonomous Driving 자율주행레벨 * Level 0 단순경고 Level 1 단순 ADAS Level 2 퓨전 ADAS Level 3 제한적자율주행 Level 4 완전자율주행 차간거리유지 Adaptive Cruise Control 현대제네시스 ( 15.12) 구글카 ( 개발중 ) 차간거리 / 차선유지 Smart Cruise Control 제한적자율주행 Auto Pilot 완전자율주행 상용화기능 차선이탈경고 Lane Departure Warning System 전방추돌경고 Forward Collision Warning System 차선유지 Lane Keeping Assist 자동긴급제동 Advanced Emergency Braking 차선변경 Lane Change Assist 측 / 후방경고 Blind Spot Warning System 주차보조자동주차원격자동주차 무인발렛주차 법제도 허용개선필요불허 *National Highway Traffic Safety Administration (NHTSA- 미국고속도로교통안전청 ) 기준
GOOGLE S SELF DRIVING CAR
Google self-driving car GPS LiDAR Camera Radar
Visualization of LIDAR data
Disengagements Reports Disengagements: deactivations of the autonomous mode when a failure of the autonomous technology is detected (272 cases) when the safe operation of the vehicle requires that the autonomous vehicle test driver disengage the autonomous mode and take immediate manual control of the vehicle. (13+56 cases)
California Autonomous Testing Disengagements (2015) https://www.wired.com/2017/02/california-dmv-autonomous-car-disengagement/
TESLA S AUTOPILOT
https://www.tesla.com/
Tesla vs Google
Google 과 Tesla 의 자율주행자동차기술차이 Computer Vision LIDAR 사용 ( 높은위치인식능력 ) Camera 사용테슬라의 CEO 앨런머스크는구글의 LIDAR 센서에관해 그렇게비싼센서를사용한자율주행자동차를개발하는것은과하다 (overkill) 라고비판 Car Control 완전자율주행기술목표 2013 년구글은일부직원의출퇴근에자율주행차를타도록했는데차안비디오카메라를모니터링한결과운전자가잠이드는등운전에집중하지않음. 이후완전자율주행기술을목표로개발 Autopilot 기능제공자동차의비행기화 비행기에서돌발상황에만파일럿이개입하듯돌발상황에서운전자의조작이필요
http://www.hankyung.com/news/app/newsview.php?aid=201607018385g
How the Accident happened
Traffic Fatalities Total miles driven in U.S. in 2014: 3,000,000,000,000 (3 million million) Fatalities: 32,675 (1 in 90 million) Tesla Autopilot mile driven since October 2015: 300,000,000 (300 million) Fatalities: 1
UBER
Pittsburgh, your self-driving Uber is arriving now
UBER/nuTonomy
NVIDIA S DRIVERWORKS
NVIDIA
Perception
Visualization
Planning
COURSE OVERVIEW
Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) Localizati on GPS, SLAM Self Driving Perception Pedestrian detection (HOG+SVM) Detection/ Segmentat ion/classif ication Dry/wet road classificati on ADAS Planning/ Control Optimal control End-toend Learning End-toend Learning Tasks Driver state Behavior Prediction/ Driver identificati on Vehicle Diagnosis Smart factory DNN * * Reinforcement * Unsupervised * *
ARTIFICIAL NEURON
ADAS Tasks Self Driving Localizati on Perception Planning/ Control Driver state Vehicle Diagnosis Smart factory Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) DNN Reinforcement GPS, SLAM Pedestrian detection (HOG+SVM) Detection/ Segmentat ion/classif ication Dry/wet road classificati on Optimal control End-toend Learning End-toend Learning Unsupervised
뉴런 : 신경망의기본단위
인공뉴런 (Artificial Neuron) 실제뉴런 뉴런의수학적모델
예시 : 연어와농어의구별 폭 (w) 7.3l + 3.4w = 100 밝기 (l) l w 농어 연어 7.3l + 3.4w 100 7.3l + 3.4w < 100
4 2 2 4 예시 : 연어와농어의구별 l 7.3 l 1.0 0.8 w 3.4 w Σ 0.6 0.4 0.2 연어 / 농어 100
Artificial Neuron w T x g(w T x) Activation function (non-linear)
Multi-layer Perceptron
TYPES OF MACHINE LEARNING Neural Network 기반방법을중심으로
Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) Localizati on GPS, SLAM Self Driving Perception Pedestrian detection (HOG+SVM) Detection/ Segmentat ion/classif ication Dry/wet road classificati on ADAS Planning/ Control Optimal control End-toend Learning End-toend Learning Tasks Driver state Behavior Prediction/ Driver identificati on Vehicle Diagnosis Smart factory DNN * * Reinforcement * Unsupervised * *
Why neural networks? Universal function approximator
Why neural networks? It can learn from data.
Why neural networks? There can be lots of variations (layouts)
Types of Machine Learning Supervised Learning Classification/Regression Semi-supervised Learning/Weakly supervised Learning/ Unsupervised Learning Clustering Feature Learning Generative Model Learning Reinforcement Learning Q-Learning Policy Gradient Learning
Supervised learning workflow
Supervised vs unsupervised Supervised Learning Data: (x, y) x is data, y is label Goal: Learn a function to map x -> y Examples: Classification, regression, object detection, semantic segmentation, image captioning, etc Data: x Unsupervised Learning Just data, no labels! Goal: Learn some structure of the data Examples: Clustering, dimensionality reduction, feature learning, generative models, etc
Unsupervised Learning Generative Model (Generative Adversarial Network)
Unsupervised Learning Generative Model (Generative Adversarial Network)
Unsupervised Learning Dimension Reduction/Feature Learning (Auto-Encoder)
Supervised vs Reinforcement Supervised Learning Data: (x, y) x is data, y is label Goal: Learn a function to map x -> y Examples: Reinforcement Learning reward x i, y i = (, ) Game state Joystick control
Reinforcement Learning Reinforcement learning is an area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward.
Reinforcement Learning
DEEP LEARNING
Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning based method Supervised SVM MLP CNN RNN (LSTM) Localizati on GPS, SLAM Self Driving Perception Pedestrian detection (HOG+SVM) Detection/ Segmentat ion/classif ication Dry/wet road classificati on ADAS Planning/ Control Optimal control End-toend Learning End-toend Learning Tasks Driver state Behavior Prediction/ Driver identificati on Vehicle Diagnosis Smart factory DNN * * Reinforcement * Unsupervised * *
구글트렌드 : 딥러닝
Neural network Back propagation, Nature 1986 장점 일반적인문제에적용할수있는학습법 Biological 시스템과관련이깊음 문제점 Training 이쉽지않음 현실적인문제에잘동작하지않음
Neural network Back propagation, Nature Non-linear SVM 1986 1992 다양한시도들 Flat structure SVM, Boosting, Biological 시스템과거리가생김 특정한문제를해결하는특정한방법 (SIFT, LBP, HOG, GMM-HMM)
Neural network Back propagation, Nature Deep belief network, Science 1986 1992 2006 비지도학습을이용한 pre-training Training 방법향샹 Dropout, RectLinear, Normalization, 컴퓨터구조의발달 GPU Multi-core computer 시스템 빅데이터 +
Neural network Back propagation, Nature Deep belief network, Science Speech 1986 1992 2006 2011
음성인식성능
Neural network Back propagation, Nature Deep belief network, Science Speech Object recognition 1986 1992 2006 2011 2012 Submission Method Error rate Supervision Deep CNN 0.16422 ISI XRCE/INRIA OXFORD_VGG FV: SIFT, LBP, GIST, CSIFT FV: SIFT and color 1M-dim features FV: SIFT and color 270K-dim features 0.26172 0.27058 0.27302
ImageNet Large Scale Visual Recognition Competition (ILSVRC) Steel drum Output: Scale T-shirt Steel drum Drumstick Mud turtle Output: Scale T-shirt Giant panda Drumstick Mud turtle http://www.image-net.org/challenges/lsvrc/
Neural network Back propagation, Nature Deep belief network, Science Speech Object recognition 1986 1992 2006 2011 2012 2013 IMAGENET 2013: 영상인식 RANK Name Error rate Description 1 NYU 0.11197 Deep Learning 2 NUS 0.12535 Deep Learning 3 OXFORD 0.13555 Deep Learning
Neural network Back propagation, Nature Deep belief network, Science Speech Object recognition 1986 1992 2006 2011 2012 2013 2014 IMAGENET 2013: 영상인식 RANK Name Error rate Description 1 Google 0.06656 Deep Learning 2 Oxford 0.07325 Deep Learning 3 MSRA 0.08062 Deep Learning
Neural network Back propagation, Nature Deep belief network, Science Speech Object recognition The game of GO 1986 1992 2006 2011 2012 2013 2014 2016
The AI race is on IBM Watson Achieves Breakthrough In Natural Language Processing Facebook Launches Big Sur Baidu Deep Speech 2 Beats Humans Google Launches TensorFlow Toyota Invests $1B In AI Labs Microsoft & U.Science & Tech, China Beat Humans on IQ
Deep Learning: Representation Learning
The Mammalian Visual Cortex is Hierarchical
Deep Learning: Scalable Machine Learning
딥러닝모델의특징 다층구조 (multi layer) 신경망의구조모사 상위층으로갈수록추상화된정보가학습과정에서자동으로생성 문제해결과정자동화 End-to-end learning 사람의개입을배제하고오직 raw input 과 output 사이에모든과정을데이터에서학습하는방향추구 분산표현 Distributed representation 여러뉴런이협력하여정보저장 / 처리
DRAWBACKS
Current Drawbacks Big data: inefficient at learning from data Supervised data: costly to annotate real-world data Need to manually select network structure Need to hyper-parameter tuning Learning rate Loss function Mini-batch size Number of training iterations Momentum Optimizer selection Defining a good reward function is difficult
Faulty Reward Functions in the Wild