AI 2017,,,,,!2

(Yoav Shoham) ( ) Stanford University (Raymond Perrault) SRI International (Erik Brynjolfsson) MIT (Jack Clark) OpenAI (Calvin LeGassick)!3

AI. AI. Yoav Shoham, Raymond Perrault, Erik Brynjolfsson, Jack Clark, and Calvin LeGassick, "The AI Index 2017 Annual Report, AI Index Committee of the One Hundred Year Study on Artificial Intelligence (AI100), Stanford University, Stanford, CA, November 2017. Korean translation by Sanghun Lee, Donghun Lee. Doc: http://cdn.aiindex.org/ai- Index-2017-Korean-Translation.pdf. Accessed: April 26, 2018. (c) 2017, AI 2017-4.0 (CC BY-ND 4.0). https://creativecommons.org/licenses/by-nd/4.0/deed.ko (c) 2017 by Stanford University, The AI Index 2017 Annual Report is made available under a Creative Commons Attribution-NoDerivatives 4.0 License (International) https://creativecommons.org/licenses/by-nd/4.0/deed.en!4

AI 2017 7 9 11 11 11 13 16 18 18 19 20 23 25 (GitHub) 25 27 27 28 (Vision) 28 (Object Detection) 28 (Visual Question Answering) 29 (Natural Language Understanding) 30 (Parsing) 30!5

(Machine Translation) 31 (Question Answering) 32 (Speech Recognition) 33 (Theorem Proving) 34 (SAT Solving) 35 (Derivative Measures) 36 39 44 47 75 77 A: 79!6

AI 2017 (AI),,.,,..!7

,.. 100 (One Hundred Year Study on AI at Stanford University, AI100) AI. AI. AI,.,,. AI (aiindex.org). AI.,,. AI.!8

AI.,,. (Data Sections). (Volume of Activity) (Technical Performance) (Derivative Measures) (Towards Human-Level Performance?)...,...., AI (AI Vibrancy Index). AI (AI Vibancy Index).!9

... (Discussion Sections),..,., (R&D)..., AI. AI..,.,,.!10

A1 SCOPUS (Artificial Intelligence). 9x 1996 9.!11

1996.,,.. 1996 6 9.!12

A2. (Machine Learning) (Articifial Intelligence),. 11x 1996 11. : 2016.!13

.. : 90..!14

: 90...!15

A3. (2016 1000 ) (2016 1000 ).. : 1980..!16

.!17

A4. 14x 2000 14.!18

!! A5 (VC). 6x 2000 6.!19

A6 (Indeed) (Monster).. Indeed. 2013 1 Indeed. 4.5x 2013 4.5.!20

Indeed. :, Indeed 5% 27%.!21

Monster. :..!22

A7.!23

.!24

(GitHub) A8 (GitHub) (TensorFlow) (Scikit-Learn). TensorFlow Scikit-Learn. GitHub,, (Star)..!25

GitHub. : (GitHub) (Fork) ( Fork ). Fork.!26

A9 ' (Artificial Intelligence)'.!27

(Vision) (Object Detection) A10 (Large Scale Visual Recognition Challenge, LSVRC). 2.5% 2010 28.5% 2.5%.!28

(Visual Question Answering) A11 (Open-ended answers). : VQA 2.0 VQA 1.0, VAQ 1.0.!29

(Natural Language Understanding) (Parsing) A12 (syntactic structure).!30

(Machine Translation) A13 -, -.!31

(Question Answering) A14.!32

!! (Speech Recognition) A15.!33

(Theorem Proving) A16 (Automatic Theorem Provers) (theorem proving problems) (Tractability)... : (solver), (solver).!34

(SAT Solving) A17 SAT (Solver).!35

(DERIVATIVE MEASURES). AI. AI. (AI Vibrancy Index). -.,.,. 2000 (normalize)..!36

: 2016,. A2. ( ). 2010 2013..!37

AI (The AI Vibrancy Index) AI (,, ) (liveliness). AI 2000,. (, AI ), AI.!38

.. 1970.,,..,.,,.,..,..,,.!39

.,. 1980 1980 (Kai-Fu Lee) (Sanjoy Mahajan) (Bayesian) BILL. 1989 BILL 56 8. 1997 (Logistello) 6 6. 1995 1952 (Arthur Samuels). 1995 (Chinook).!40

1997 1950 1967, 1997 IBM (DeepBlue) (Gary Kasparov).. 2011! (Jeopardy!) 2011 IBM (Watson)!(Jeopardy!) 1 (Brad Rutter), (Ken Jennings) 100. 2015 (Atari) 2015 (DeepMind) (reinforcement learning system) 47. (Montezuma's Revenge), (Breakout). 2016 (ImageNet) (ImageNet) (labeling) (error rate) 2010 28% 2016 3%. ( ) 5%. 1.!41

2016 2016 3 (DeepMind) (AlphaGo) 9 4 1. 2017 3 (DeepMind) (Alpha Go Master), 1 9 3 0. 2017 10 (DeepMind) < > (AlphaGo Zero), 9 4 1 100 100. 2017 (Skin Cancer Classification) 2017 < > (Andre Esteva) 2,032 129,450, 21.. 2017 (Switchboard) 2017 IBM (NIST Switchboard) (human parity).!42

2017 2017 1 (CMU) (Libratus) 2 (Head's Up No-Limit Texas Hold em) 12. 2017 2 (DeepStack) 11 3 (DeepStack) ' '. 2017 (Ms. Pac-Man) (Maluuba) 2600 999,990. 2.!43

,.,,... AI. AI. (,, )., (commonsense reasoning). (, )..,.., 5,.!44

AI (optimistic bias).,... AI,..,. (sex), (race), (gender), (ethnicity), (sexuality).??..!45

AI., (R&D)..,..,,,... AI..,,,,.!46

100% (bias). AI,,.,,. (Barbara Grosz) (Harvard) AI, AI. ( ),.. 2016 (IJCAI- 16) AI(human-aware AI), 2018 (AAAI-18) (human-ai collaboration). AI(human-inthe loop AI) AI.!47

.,, ( ). (utterance).,..., (IRB).,.,.. AI ( 3 : ).!48

. AI,.!49

(Eric Horvitz) (Microsoft) AI. AI 100 (AI100), 100 AI100. AI100 (connected presence). AI 2015 AI100. AI.,..,,, (Convolutional Neural Networks) (variants).,. AI (AI Vibrancy Index),.,.!50

. ( ) (,,,, ).. (commonsense reasoning). (commonsense understanding)..,,. AI.,.,.. AAAI 2017,,.!51

..,.,. AI.,.!52

(Kai-Fu Lee) (, Sinovation Ventures) AI.. ( ). " "..?,.. 50. 10. (Mobike) 2 10. 2 GPS, 20. 3.,. 3..!53

? 15.., Face++ 3,,,. 2030. (iteration). 2017 7 2030.., ', ( 众创业众创 )' 4..,,.,. 4!54

(Alan Mackworth) (University of British Columbia) AI... 5..... EU. /. EU.,. (data source),. 5.!55

.,.?,? AI2, OpenAI, WEF Turing Institute. 1) / 2) / ( [CRA] ).., CRA.,... / / /.. (, CAPTCHA). (Multi-Agent Systems, MAS).!56

(RoboCup) 6 Trading Agent Competitions(TACs). (Loebner Prize) 7 (Turing Test) 8... ( ). (meta-ai).,,,, ( ),,,,,,,.? -? 6 1996. 7,.. 8 1950 (Alan Turing),.!57

(Andrew Ng), (Coursera, Stanford) 19. 19. AI.... AI.,.,. '..!58

,.,....,.,.!59

(Daniela Rus) MIT(Massachusetts Institute of Technology),... 3D....,,,.,.. (MOOC),.,.!60

..,....,., 3..,.,,. AI.,,,.,. AI. AI!61

, (framework).,.,,. AI.!62

(Megan Smith) 3 (CTO), Shift 7 (Susan Alzner) UN (UN Non-Governmental Liaison Service) AI 2017. AI.. AI..,.!63

.,..,,...,,. /.,.,,,..,..!64

2016 - (OSTP) / (OSTP) (CTO) (NSTC), 2016. 2016 10. -. 2016 5 3, CTO -,. 2016 10 12, CTO, (CTO)!65

,..,,, /.? / (,,,, ).,.... /.,, (malicious iterations). '.... ' '..!66

2030 (2030 Agenda for Sustainable Development & SDGs) (Universal Declaration of Human Rights) (Declaration on Rights of Indigenous Peoples) (Beijing Declaration and Platform for Action on women s human rights) (Convention on the Rights of Persons with Disabilities), (International Covenant on Economic, Social and Cultural Rights) - 1 (IEEE Ethically Aligned Design General Principles - esp Principle 1 on Human Benefit lists (p16) 2017 MIT CEO..,..,,..,..,..!67

(The Algorithmic Justice League, www.ajlunited.org/). UN. (The Computer Science for All Movement, CSforAll - csforallconsortium.org/). AI(The AI4All initiative, ai-4-all.org/),,,. / /. / / ( [TQ in Public Policy] ). AI.. (Cathy O Neil) 2017 11 14 ( The Ivory Tower Can t Keep Ignoring Tech ).. AI.!68

AI,.!69

(Sebastian Thrun), (Stanford, Udacity). 60.... (Deep Mind) (AlphaGo)... (safety driver). (Cresta)......!70

.. FarmersFeedUS.org 155. 2% 98%. 75%,,,...,.?..... 1/9.......!71

.,.!72

(Michael Wooldridge) (University of Oxford) AI, (International Joint Conference on AI, IJCAI, http://www.ijcai.org/), (European Association for AI, http://www.eurai.org/).,,,..,,.. (1996-2001 ),.,.. ( http://tinyurl.com/y9g74kkr ).,.!73

,. ( ),.,,. (2027 AI ).. (technical performance) ( ). (general AI)...,. (turing test)..?.,.!74

.. AI. AI 2017.,. (@indexingai).. AI AI. AI,,...!75

AI... PDF.... AI.. @indexingai feedback@aiindex.org. AI aiindex.org.!76

AI " "(AI100). Tom Abate, Amy Adams, Russ Altman, Tiffany Murray, Andrew Myers.,, Bytedance(, Toutiao). AI. AI.. Michael Bowling, Ernie Davis, Julia Hirschberg, Eric Horvitz, Karen Levy, Alan Mackworth, Tom Mitchell, Sandy Pentland, Chris Ré, Daniela Rus, Sebastian Thrun, Hal Varian, and Toby Walsh... AI. Susan Alzner, Barbara Grosz, Eric Horvitz, Kai-Fu Lee, Alan Mackworth, Andrew Ng, Daniela Rus, Megan Smith, Sebastian Thrun, Michael Wooldridge..!77

Toby Boyd, Kevin Leyton-Brown, Miles Brundage, AJ Bruno, Jeff Dean, Catherine Dong, Peter Eckersley, Stefano Ermon, Oren Etzioni, Carl Germann, Marie Hagman, Laura Hegarty, Holger Hoose, Anita Huang, Dan Jurafsky, Kevin Knight, Jure Leskovec, Tim Li, Terah Lyons, Mariano Mamertino, Christopher Manning, Gary Marcus, Dewey Murdick, Lynne Parker, Daniel Rock, Amy Sandjideh, Skyler Schain, Geoff Sutcliffe, Fabian Westerheide, Susan Woodward.. Allen Institute for Artificial Intelligence, Crunchbase, Electronic Frontier Foundation, Elsevier, EuroMatrix, Google Brain, Indeed.com, Monster.com, Sand Hill Econometrics, Sinovation Ventures, TrendKite, VentureSource.. Chitta Baral, Maria Gini, Carol Hamilton, Kathryn B. Laskey, George Lee, Andrew McCallum, Laurent Michel, Mary Ellen Perry, Claude-Guy Quimper, Priscilla Rasmussen, Vesna Sabljakovic-Fritz, Terry Sejnowski, Brian Williams, Ramin Zabih.. Lance Fortnow, Charles Isbell, Leslie Kaelbling, Steven Lavalle, Dan Klein, Lenny Pitt, Mehran Saham, Tuomas Sandholm, Michael-David Sasson, Manuela Veloso, Dan Weld...!78

A: A1. Elsevier Scopus 7 (69,794,685). Scopus. Scopus (Computer Science) (Artificial Intelligence) (key term). Scopus (Computer Science) (Artificial Intelligence) (key term) 200,237. Scopus (Computer Science) 5 (4,868,421). Scopus 2017 11.!79

Scopus (query),,. 2000. title-abs-key(artificial intelligence) AND SUBJAREA(COMP) AND PUBYEAR AFT 1999 AND PUBYEAR BEF 2001 SUBJAREA(COMP) AND PUBYEAR AFT 1999 AND PUBYEAR BEF 2001 Scopus PUBYEAR AFT 1999 AND PUBYEAR BEF 2001 1996 2016. Elsevier Scopus API Scopus. Scopus Scopus. Elsevier API Elsevier API. Scopus API API.!80

Scopus. Scopus. Scopus SUBJAREA (COMP) AND PUBYEAR BEF 2000. Elsevier 1995 Scopus 1995. Scopus 1996. Scopus. Elsevier. Web of Knowledge, Microsoft Academic, DBLP, CiteSeerX and Google Scholar.,.!81

A2... University of California Berkeley, Carnegie Mellon University, Georgia Institute of Technology, University of Illinois Urbana Champaign, Massachusetts Institute of Technology, Stanford, and University of Washington... AI.....!82

. 2015-2016. 1. 2015-2016 ( ).. ( ) 2016-2017 2015-2016. 2016-2017. 2015-2016 2016-2017 (smoothing),.!83

A3... AAAI, AAMAS, ACL, CP, CVPR, ECAI, ICAPS, ICRA, ICLR, ICML, IJCAI, IROS, KR, NIPS, UAI.. 2016 1000 2016 1000. AI.....!84

A4. (Crunchbase) (VentureSource), (Sand Hill Econometrics),. Crunchbase. Crunchbase. Crunchbase API. VentureSource. Crunchbase VentureSource AI. VentureSource. VentureSource.!85

Crunchbase API Crunchbase VentureSource Sand Hill Econometrics. Crunchbase (Artificial Intelligence) (Machine Learning) (Natural Language Processing) (Computer Vision) (Facial Recognition) (Image Recognition) (Speech Recognition) (Semantic Search) (Semantic Web) (Text Analytics) (Virtual Assistant) (Visual Search) (Predictive Analytics) (Intelligent System)..!86

A5. (Crunchbase) (VentureSource), (Sand Hill Econometrics),. A4 ( ). VentureSource,. VentureSource Sand Hill Econometrics.!87

A6. (Indeed.com) 9 (Monster.com) 10 Indeed.com 2013 1. Monster.com..,.. AI Indeed Monster. Indeed Monster. Indeed 50%.. 9 10!88

(Artificial Intelligence), (Machine Learning), (Natural Language Processing). 90%. Indeed. Indeed 2013 (normalized). CEB11 TalentNeuron Monster 2015, 2016, 2017(11 10 ). TalentNeuron. 11 Fortune 500!89

A7. (International Federation of Robotics, IFR) (World Robotics Report).. ISO 8373:2012 standard. (International Federation of Robotics, IFR) (World Robotics Report). AI 2000...!90

A8. (GitHub) (Github) GitHub Archive GitHub Archive on BigQuery (repositories). apache/incubator-mxnet, BVLC/caffe, caffe2/caffe2, dmlc/mxnet, fchollet/keras, Microsoft/CNTK, pytorch/pytorch, scikit-learn/scikit-learn, tensorflow/tensorflow, Theano/Theano. GitHub archive Google BigQuery. AI BigQuery repository WatchEvents. 2016.!91

SELECT project, YEAR(star_date) as yearly, MONTH(star_date) as monthly, SUM(daily_stars) as monthly_stars FROM ( SELECT repo.name as project, DATE(created_at) as star_date, COUNT(*) as daily_stars FROM TABLE_DATE_RANGE( [githubarchive:day.], TIMESTAMP( 20160101"), TIMESTAMP( 20161231")) WHERE repo.name IN ( tensorflow/tensorflow", fchollet/keras", apache/incubator-mxnet", scikit-learn/scikit-learn", "caffe2/caffe2", "pytorch/pytorch", "Microsoft/CNTK", Theano/Theano", "dmlc/mxnet", "BVLC/caffe") AND type = WatchEvent GROUP BY project, star_date ) GROUP BY project, yearly, monthly ORDER BY project, yearly, monthly!92

GitHub Archive repository.. GitHub repository. GitHub. star-history. GitHub Fork. GitHub Fork. Fork aiindex.org. BigQuery type= WatchEvent type= ForkEvent.!93

A9. TrendKite: PR. TrendKite,,. ( ). AI. (disproportionate). Query Artificial Intelligence AND NOT MarketIntelligenceCenter.com s" NOT site_urls_ll:( individual.com" OR "MarketIntelligenceCenter.com") TrendKite..!94

..!95

A10. LSVRC ImageNet Competition 2010-2017 ImageNet Data Set 2010 LSVRC ImageNet. (Metrics) LSVRC. ImageNet LSVRC. ImageNet 2017. ILSVRC. Russakovsky et al, 2015.!96

A11. (VQA) (VQA) Arxiv ( ) VQA Data Set VQA, 10. ( ). VQA.. 2016 2017 VQA 1.0.,. (Ensembles).!97

ImageNet AI. ImageNet', VQA. VQA 1.0 VQA 2.0.!98

A12. (Parsing) (parsing) Penn Treebank Penn Treebank (constituency-based parse tree). 23 (automatic parsers). (automatic parsers) (parse) GOLD parse. (precision) (recall) F1 (F1 score). Penn Treebank 23 (parser) F1. F1 40.. (parser). 1995 (parser) F1. (Ensembles).!99

(automatic parsers) (parser) 40 100. 40 (corpus) F1.!100

A13. / (Workshop on Machine Translation, WMT) EuroMatrix (WMT) (Conference on Machine Translation). WMT.. WMT.. BLEU, BLEU. BLEU (precision) 0 1. (corpus) BLEU. -, - BLEU. WMT BLEU.!101

EuroMatrix 2006 -, - (task) BLEU. BLEU. (tokenizing) BLEU(11b). -, - BLEU. 11b. BLEU. BLEU (corpus) BLEU., 2016 2017 BLEU (2017 2015 ). 2016. BLEU.!102

A14. Stanford Question Answering Dataset / (SQuAD, The Stanford Question Answering Dataset) 500 (articles) 100,000 -.. SQuAD Exact Match(EM). Exact Match. SQuAD (state of the art) EM. SQuAD. SQuAD (quotes)..!103

SQuAD. SQuAD 2016 6 (task). SQuAD Exact Match 82.304.!104

A15. Switchboard Hub5 00 ( ). EFF AI Switchboard Hub5 00 (Word Error Rate, WER). (,, ). (Word Accuracy), 1-WER. (Electronic Frontier Foundation, EFF) Hub5 00.. Switchboard. (overfit),.!105

Switchboard Hub5 00 WER. 5.1%, 5.9%, 5%. 5.1% WER.!106

A16. Thousands of Problems for Theorem Provers (TPTP) TPTP. (The Automatic Theorem Proving, ATP) (ATP). TPTP (subset), 2010 TPTP v5.0.0 TPTP. (tractability). TPTP (maintainer) TPTP TPTP TPTP. TPTP. aiindex.org.!107

TPTP (quirk). ATP. ATP. (maintainer) TPTP v6.4.0.!108

A17. (SAT) SAT SAT SAT (Satisfiability, SAT). (Holger Hoos) (Kevin Leyton-Brown) 2007 69 (solver) 1076 solver. 2007 SAT solver solver. (Holger Hoos) (Kevin Leyton-Brown) solver. AI. (Holger Hoos) (Kevin Leyton-Brown),.!109

SAT solver, SAT solver (novelty)... AI SAT solver.!110

!111