as well as want to go back. he does a good job of capturing both pieces of that so those are the two books that are front and center for me right now. >> bottini wants to know what you are reading. send us your summer reading list @booktv on twitter, instagram or on facebook. booktv on c-span2, television for serious readers. >> i am ready to get started. if people out having breakfast could come and join us, that would be great. we have folks coming up from the answer way downstairs. i realize that and early morning tech event isn't oxymoron so i apologize for the timing but thank you for coming. my name is nick, partner at [inaudible] and i spent much of the last five years trying to

figure out the economic artificial intelligence and i have to confess i am not yet there. i'm still working on it. our speaker today 20 has been a lot of time on the subject and published academic papers and written a book on the subject that he will talk about today. i've had the opportunity to work with him on multiple projects and it was particularly interesting is he is and distinguish academic and a successful entrepreneur in the field of artificial intelligen intelligence. he's a professor of entrepreneurship at the university of toronto where he focuses on the economics of artificial intelligence and a research associate at the national bureau of economic research in boston and on the entrepreneurial front he's founded three entities associated with artificial

intelligence that has been interesting. one was about to open up operations in new york but first the one that would open operations in new york is called creative destruction lab and i've been advisor to the organization started in toronto and it is an accelerator that is home to 150 ai driven startups for those beginning track is the largest concentration on the planet so interesting achievement and the expanding operations into new york. the next is ai which is a program for young ai entrepreneurs in canada and the third is kindred, the startup the build machines with humanlike intelligence and was originally named the mit tech reviews that the smallest companies and to see inside ai 100. this is due in part to the work

in ai in 2017 which was a commercial gaining newspaper named them one of the 50 most powerful people in canadian business and today he will talk about his new book, prediction machines, the simple economics of artificial intelligence it was published worldwide this week by harvest business review press so here is how the logistics will work today. turning will speak for about one hour and then we have 15 minutes of q&a and then will have coffee in discussion for everyone together outside afterward and as you probably saw there's a big stack of books in the back, the free to take one on your way out and then please try me in welcoming ajay agrawal. [applause] >> thank you, nick. thank you for hosting this.

i don't know if kelly is here but she coordinated my logistics so i like to thinker. thank you to jim and beth fisher who are our literary agents and helped us bring this book to fruition. just so i can calibrate how many people here have an ai company? okay. entrepreneurs? in terms of familiarity if you had to stand up now or would feel comfortable standing up and in a few sentences defined what ai is? three or four. those with organizations if you were to characterize where you would please your organization

in having an ai strategy category one we have a strategy in the works and category one and category two is we are putting a team together to start working on that and category three is we have no idea where to start and those in category one, show of hands -- one, two, three people. category two? okay and category three? all right. hopefully, by the end of this at least the first question which is defined and everyone will leave the room feeling comfortable to define it and hopefully you will have a sense of where to start.

i don't need to tell this audience any both selected to be here that ai is i think the reason it's a fair amount of enthusiasm with this technology is it's what they will call a general-purpose technology that is everywhere. it's not hard to find any market where there is not some ai applications that are rolling in to advance the productivity whatever they are doing. at the same time there's a fair amount of angst and people can figure out what it is and the indications and how they can deploy it and what that means for humans. my role this morning and we try to compass with the book is to remove some of the anxiety by way of providing clarity. how can we think of ai particular those who don't have computer science background and to do that what we do is take a

topic that is the domain of computer science and the layer of a different lens on it from a discipline that we often don't associate with clarity and that is economics. by putting in economics later on top of a computer science background, in this case, we think we can provide a fair amount of insight into the development of ai. my background is i'm and apply the economic professor at the university of toronto and i teach business school and two blocks down for my building is a computer science building. due to some serendipity and a hiring mistake 25 years ago toronto became the epicenter of the recent renaissance in ai.

today several of the most powerful industrial ai groups in the world are headed by people who are ten years ago, at toronto. the person who was originally led the ai group here in new york at facebook was ten years ago at the university of toronto. the person believes ai research at apple in cupertino, ten years ago, university of toronto but the person believes the ai research for elon musk ten years ago, university of toronto. the reason that is relevant is that i am the founder of a program in toronto that nick mentioned that his opening here under the professor that is here called the creative destruction lab. the focus of the lab is to transition the science projects into massively scalable

companies. five years ago what started as a trickle and turn into a flood was graduate students coming out of the slab and the companies are coming from around the world to create destruction lab but out of this lab on our campus the first one the game was a graduate student who said i will use this new technique for predicting which molecules will most efficiently find for the purpose of self-discovery. right after a not him another one came up the that i will use this technology for predicting which credit card transactions are fraudulent versus legitimate. another one came and said i would use this technology to look at medical images in predict what tumors are benign versus malignant.

another one came as i will use this technology to protect which automobiles have defects before they will off the production line. and so on. as we were sitting in the lab myself into my colleagues it did not take a rocket scientist to figure out there something unusual coming out of this computer science. the same underlying technique which we now refer to as deep learning and reinforcement learning in the same underlying technique is being used to give a material lift to this very wide set application. after a while we just wanted to document what we're seeing. then we compiled her notes into what are the general lessons to be learned that are applicable across all of the cases.

what i will do now is give you some of the key points in my book which at the stores yesterday. i don't need to say to this audience that a lot of people are feeling that today in ai field a lot like 19 and if i felt the internet. heidi, if we could give me a little echo. it feels a lot like 1995 without the internet. most people remember the 1995 was a real transition year or the internet. we had the internet for at least a couple of decades before 95 was used by the military and academics and it was growing over time and then in 1995 there was a big jump and that was the year that bill gates wrote to his famous tidal wave e-mail and microsoft launched 95 and

[inaudible] in august of that year netscape went public, 3 billion-dollar ipo for a company that had almost generated no profits. right after this, 96, early 97 the language around the internet started to change. we stopped referring to the internet as new technology and instead began referring to it as a new economy. the point was that it had permeated sony parts our lives that we stop thinking as the technology and started digging about it as a different way of interacting in the economy. ceos and journalists and interviewers and investors and politicians all started referring to this as a new economy. everybody referring to the new

economy except for one group of people and that was economists. economists said wait, this is not a new economy but this is exactly the same economy we've already had and in fact, we won't have to change a single word single page in an economic textbook. all the underlying models still hold. everything is still driven by supply and demand, production and consumption, prices and cost and it is all the same. the only thing that has changed is that the relative cost of a few key inputs have fallen dramatically. the cost of digitally disturbing goods and services and the cost of search and the cost of medication and that is the way economists be the world. the thing that economists are good at is to take new technology and strip all the fun and was a dream out of it and

resolve a tech down to a single question and the question is what does this reduce the cost of? surprisingly that often gives us great insight. for example, the heart of silicon valley is the semi conductor industry. if you ask a computer scientist or an electrical engineer could you please describe to me the rise of the semiconductor industry? in their heads they will have an image like this they will describe to you the underlying science behind cramming more transistors onto a chip. bill explain how it doubles every 18 months and the temptations of more law and that is what that technologist looks like to you. if you ask the same question to economist can you describe to me the rise of the semiconductor industry is not the immense they have in their head. they will have this image.

the reason that economists will say the semiconductors are so foundational while there are many things happening in the bay that semiconductors are the engine of the innovation economy in silicon valley is because the thing for which the cost fell such a foundational input in the case of semiconductors was arithmetic. economists think of semiconductors as the rise of semiconductors dropping -- how many people saw the film hidden figures? does anyone remember what the job title of the women who were the protagonists in that film? computers. they were called computers because what they did was they came to work and computed. there is a scene in the film with a role in the big machine and the big new computer and everyone is trying to figure out what that means for them. okay. when the cost of something false, three important things happen. by understanding these three

things we gain great insight into figuring out how this will affect the economy. we start off by going back to your economics 101 and just the very thing everyone learns is downward slope demand curves and when something gets cheaper we use more of it. for example, things for which we already used arithmetic and things like you have the types of decorations they were doing in nassau in the film and census bureau and the military where we used a lot of written project we started using a lot more and so it was better, faster cheaper arithmetic and we did of it. the cost falls and we use more of it. this is where things become

interesting is we use more of it not just things that were traditionally arithmetic problems but we take things that were not arithmetic problems and convert them into arithmetic problems to take advantage of the new cheap arithmetic and an example of that is photography. photography used to be chemistry problems and we saw photography with chemistry and made film. as arithmetic became cheap we transitioned to arithmetic based solution to photography and music and communication and banking. one thing after another we transitioned to an arthritic based solution. now onto ai. if we were to ask a technologist and an engineer to please describe to me the rise of ai what they described you is the rise of the science and the statistics underlying neural networks. they would talk about inputs and outputs and the weights of those links and they would explain to you the development of the task [inaudible] if you were to,

instead, ask an economist could you please describe to me the rise of ai they would not have this image in their head but this image. at this point people will be thinking wait, economists are reasonably single-minded and they see everything the same and at some level that is true but this is the key to understanding why economists think of ai as an category of its own. if you go to the consumer electronics show in las vegas and you see a rainbow of new technologies and robotics and drones and artificial intelligence and virtual reality and so on is he so many areas of tech and you say there's so many things so why is ai so special? the reason the economist will say ai is in a different category than all those other things and the reason is because the thing for which the cost falls as a foundational input

into such a wide range of activities that we conduct. of course, in the case of ai that means production. we can think of ai as the rise of ai in the dropped of cost of production. anytime you read a magazine article or the something about ai replace it with the words cheap fiction in the article will suddenly seem less magical and more practical. you can make sense of what they tried to do. first of all, how do we define fiction? fiction, we defined, as taking information you have to generate information you do not have. that includes what most people would traditionally call fiction, like demand forecastint five years of sales to protect or forecast next quarter sales so that is an obvious form of prediction but a less obvious

form that we would still call protection is classification so for example, i mentioned, looking at a medical image that is the information we have for the pixels in the image and the information do not have is whether the tumor is benign or malignant. the ai generating that classification we would call fiction. prediction is taking information you have to generate information you do not have. we have assisted drop, a plummeting in the cost of production and what does that mean and what are the applications for business? ... >> better, faster, cheaper. so we'll just start seeing a.i.s coming in and replacing

the traditional statistical techniques we use for doing prediction. at the same time, we'll also -- and this is where it'll become interesting. this is where, in my view, a.i. on the business side -- separate from the computer science side -- becomes an art more than a science, which is we will start converting nonprediction problems into prediction problems. so we'll start using more prediction by converting things into prediction problems. an example of that, just like we did in arithmetic, an example of that is driving. this is the one that everyone's most familiar with. so driving, we've had acon the mouse -- autonomous vehicles for a long time. but we also deployed our autonomous vehicles in a controlled setting, a factory or a warehouse. and the way we did it, simplified version, is that an engineer would have the floor plans of, let's say, a factory or warehouse, and they would

program a robot to move around the factory floor. and then they'd give the robot a bit of intelligence. they'd put a camera on the front, and they would tell the robot if somebody walks in front, then stop. if the shelf is empty, then move to the next shelf. if, then, if, then. so a series of logic, of rules that gave the row bot some intelligence. robot some intelligence. the problem was you could never take that robot out of the controlled environment and put it in an uncontrolled environment because there were too many ifs. in fact, there's an infinite number of ifs. if it's dark, if it's raining, if a child runs up to the edge of a road, if an oncoming bar with a left blinker on. and all the ifs are interactive. in an uncontrolled environment, as recently as six years ago the experts in the field were saying we will not have an autonomous

car on a city street in our lifetime. because we simply cannot program all the ifs. until people in this, in the machine learning field reframe the problem and said is rather than programming an infinite number of ifs, what if we change the problem to the instead making one prediction. and that prediction would be what would a good human driver do. and so the simplified version of how an autonomous vehicle works, a way to think about it, is imagine a car, and in the car you put a human. the human puts a human in the driver's seat, and imagine putting an a.i. in the passenger seat. and so what we do is we tell the human who's sitting in the car, drive. just drive. drive for a million miles. and so the human sits in the car and starts to drive. and as they're driving, they have data coming in through the cameras in the front of their head and in the microphones in the side of their head, and as

the data comes in, we process the data with our monkey brains, and then we take an action. and the actions are very simple, it's a very small set of thens. we can turn left, we can turn right, we can brake, and we can accelerate. that's it. we have many, many ifs coming in, and we have a very small number of thens. okay. now on to the a.i., so imagine the a.i. is sitting beside the human. the a.i. doesn't have its own eyes or ears, so we give it cameras, radar, lidar around the car. and the way to think about it the data's flowing in as you're driving, and every fraction of a second the a.i. is looking over and trying to predict what will the human driver do in the next second. and so in the beginning, the a.i. is not a very good predicter. they have big confidence intervals on their predictions meaning they're not very accurate. and is they say i think she's going to turn left, i think

she's going to go straight, i think she's going to brake, and then something happens, either she turns left or doesn't turn left. and every time it makes a prediction and then they get to the a.i. observes what the human driver does, they were right, they double down on their model. if they're wrong, they update their model. and then they make a different prediction, potentially, the next time. okay. and so as they're driving, in the beginning the a.i.'s confidence doesn't filter very wide, they're making a lot of mistakes, but as they drive and learn and correct their mistakes, the confidence intervals get smaller, smaller, smaller until at some point the a.i. is such a good predicter of what the human driver would do that we say the a.i. can just do it its. so the a.i.'s become a prediction machine for driving. and that is really where, i think, a lot of the enthusiasm is. it's converting problems that weren't traditionally prediction problems into prediction problems to take advantage of the new, cheap prediction.

so driving is an obvious one, but we've done it in so many other areas. translation. translation used to be a rules-based problem. we had lung wises who were experts -- linguists who were experts, and they would do translations. but we've converted translation into a prediction problem. and now for those of you who, for example, use google translate even between a year ago and today, the improvement is significant, and it feels like it's not too far away that we'll have a commercial-grade translator that's based entirely on prediction rather than rules. okay. in our lab, creative destruction lab, it was mentioned this is where we have all these a.i. companies that have come in in the last five years, and as far as we know creative destruction lab is now home to the greatest conservation of a.i. start-ups as any place on earth. they're coming in from all over the place. and these a.i. companies, each

one is, you know, working on solving a prediction problem. and now we're having a lot of corporates, large companies, come in who want to better understand a.i., and an interesting group are people who are heads of h.r. so the common conversation we have is, you know, the head of h.r. of this large corporation or that corporation will fly to toronto, and they'll say we're trying to learn more about a.i., i need to know for recruiting what types of skills should we recruit for, how should we train our staff in order to prepare them for a.i.? we need to know for the other parts of our company, the sales department, the manufacturing compartment, the design compartment. but not for my compartment because i work in h.r., h.r.'s a very human business. that's what they say to us. but for the other parts of the country, we need to learn about a.i. for recruiting. now, most of you know where this is going. one by one a.i. companies are

transforming the h.r. process into a series of predictions. so what do h.r. people do? well, the very first thing they do is hire, they recruit. recruiting is effectively now a prediction problem. we get a series of resumés and cover letters and interview transcripts, and then we predict from a set of applicants which ones will be best for the job. once we've hired people, the next thing we do is promotion. what's promotion? a prediction problem. we have a set of people working in the company, and we need to predict of those people who would be best in the next level up. then our next issue is retention. let's say we're at a 5,000 or 10,000-person organization, we need to retain our best people. well, what's that? it's a prediction problem. we need to predict which of our stars are most creative and what types of incentives would be most effective to keep them. and so on. one by one these positions, these roles are being converted into prediction problems so that a.i.s can tackle them.

okay, so item number one when the cost of something falls, we use more of it. and we use more of it for both traditional things, and we also convert new things, in this case, into prediction problems to take advantage of the better, cheaper, faster prediction. here's the number two and number three. when the cost of something falls, it affects the value of other stuff. in economics language, we call that cross-price elasticity. basically what it means is you can think of things as being related to the focal thing as complements or substitutes. so think of coffee. if the cost of coffee were to fall, the complements to coffee -- the things we use with coffee like cream and sugar -- the value of those things would go up because if the cost of coffee falls, we'll start consuming more coffee, and because we are, we'll also

consume more cream and sugar, and therefore, the value of cream and sugar will go up. or if the cost of golf clubs falls, the value of golf balls goes up. all right. and so those are complements. the value of complements go up. the value of substitutes go down. so in the case of the cost of coffee falls, at the margins some people switch from tea to coffee, the value of tea the falls because the demand for tea diminishes. tea is a substitute, cream and sugar are complements. now, how does that work in a.i.? we can take any task and break it down into these components, any task, and so it doesn't matter what part of the organization you work in, there'll be a series of tasks that you do, and every task can be broken down into these bits. so, for example, as i'm giving this talk right now, i bang my knee on the lectern, and three

days from now my knee is really sore. and so i go to the doc, and i tell her my knee's sore, and so she says, okay, she asks me a bunch of questions. maybe she sends me for an x-ray. that's input. she's collecting input. then she makes a prediction. and her prediction might be i think with 90% probability you've bruised your knee. with 10% probability, there's a hairline fracture. then she applies judgment. and her judgment is, the way to think about it is how costly would it be for this patient if they actually have a bruise, but i mistakenly treat it as a fracture. versus if they actually have a fracture and i mistakenly treat it as a bruise. that's her judgment. she's taking all the things that she knows about me and is trying to figure out the cost of the mistake. that's judgment. then she takes an action. so her action might be saying,

okay, i've decided i think it's a bruise, i'm going to treat it as a bruise, put some ice on it, raise your leg, and if it's still hurt anything a week, come back and see me. that's the action. then there's an outcome. the outcome would be a week laterrer, let's say my knee's better, i'm good to go, and so we've learned that she was right, and that becomes feedback data which we use that, in this case, strengthens the model. or let's say it was wrong, a week later my leg is worse, and then that's feedback data, and we update and change the models for the next time. okay. so any task we can break into these components, and we find this very useful for designing strategy around a.i.. and for thinking through what are the implications for jobs and the economy and so on. here's how. if you look at this diagram, it's very clear what is the substitute for machine intelligence.

so as machine intelligence increases, as the cost of machine prediction falls, what's the substitute for machine prediction? it's the box in the middle, human prediction. the value of human prediction will fall as the capability of machine prediction increases. so we are quite poor predicters. we're slow, we're noisy, we have all sorts of systematic biases in our prediction. it's been very well documented in books like danny canman's thinking fast and slow, the predictably irrational. all these things document how terrible humans are at making predictions, but we still make them all the time. and so as the capabilities of machine prediction increase, the cost of machine prediction falls, the value of human prediction will plunge. so our human prediction capabilities become less and less valuable because the machines can do it so much

better, faster, cheaper. and that's the part i think the press has been fascinated with, and that's led to a mischaracterization of a complete wiping out of the role of humans. what they miss when they focus on that are all the other boxes. the other boxes are the complements. they are the cream and sugar. they're the things that will increase in value as the price of machine prediction falls. the one that the press has talked about is the first one, input. so how many people have heard the phrase data is the new oil? okay, most people. that is effectively talking about the first box, the input. we've always had data. that day we've had, a lot of that data we've had for a long time. why is it the new oil? what makes it new? what makes it new is it's way more valuable. the same data is way more valuable today than it was ten years ago. why? because the cost of prediction has fallen, and so the value of

the data has gone way up. okay? it's a complement. the data is a complement. it's more valuable now because we can do more things with it because prediction is better, faster, cheaper. so the press has done a good job of talking about the input box, but the press has not done a good job so far of describing the other bits. human judgment. a.i.s do prediction. they don't do judgment. they don't know what to do with their predictions. we have to give them guidance on what to do with the predictions. we decidement. that's judgment -- we decide. that's judgment. so, for example, the doctor who's deciding what's the cost of a mistake, she's using her judgment. and so what's interesting is that we're always applying our judgment, but the value of our human judgment goes up as the cost of prediction goes down. as we start getting better, faster, cheaper predictions, as the fidelity of our predictions increase, of machine predictions increase, the value of our human

judgment goes up because we're applying our judgment to much better predictions. okay. finish the next one is action. the predictions are generally used to inform an action. what action should we take? companies, operating companies own very often now as the incumbents they own the action. that's a valuable asset. it's not just the input, the data that matters. it's not just data's the new oil. your actions are valuable because now you'll be able to take better actions because you're basing those actions on higher fidelity and predictions. so in our has been where we have all these a.i. companies, one of the issues is that they build and sell predictions, but they don't own the actions. and so they have to sell their predictions to someone who does own the actions, because without the actions their predictions are worthless. so actions are a complement. the value of actions goes up.

where we have the most amount of negotiations with all the a.i. start-ups, a lot of them who are selling their predictions to larger enterprises, it's on feedback. that's the gold dust. when we take an action, we find out later whether that action was a good one or a bad one, and that's how we learn to update the a.i.. so all our a.i. start-ups that are trying to license their prediction outputs, sell their predictions to enterprises, are all trying to get their hands on the feedback data. and the company, some of them realize, some of them don't that that's the gold dust. that day in very -- and it's, to some ec tent, much more valuable than this data. this day data, using the oil analogy, you use it and you burn it, it's gone. in other words, you use it to train your a.i. the first time. but once you've trained your a.i. the first time, there's a caveat here, you use it to train your mod pell, and then done.

the value's gone. the ongoing value comes from the feedback data that allows the a a.i. to continue to learn. okay. so the key point here is in thinking about strategy and in thinking about the implications of jobs is that the value of human prediction falls, but the value of all these complementary assets go up. and a useful way to think about this, an example, is think about when spread sheets rolled into town and accountants. the accountants originally, you know, you could think of them as having two broad skill sets. one of them was that they would type in the numbers and add, and so they had, you know, we valued their ability to type vast and to add fast. and then the second skill they had was to ask good questions. so let's say they were doing a model to estimate the present value of some asset, and then they might ask a good question by saying, well, what would happen if interest rates went up by 1%, or what would happen if sales went up by 4% in the

fourth quarter. and so they would ask a question. but then they'd have to the, as soon as they'd ask the question, they'd have to start at the beginning and retype everything and re-add up the whole set of numbers in order to test that new scenario. when spread sheets rolled into town, the value of a human ability to type fast and add fast went down because the machine could now do that. but the value of being able to ask good questions of your data went up because it became much quicker and easier to ask a question. so if you were good at asking questions of, you know, doing scenario analysis, the value of you as an accountant went up. so if you were an accountant where your key skill was asking good guess -- good questions, your value went up. okay. so if a.i. is just prediction, if the current renaissance in a.i. is really all about loring the cost of prediction and it's not dolores from west world of c-3po, then why is there so much fuss?

why is there so much fuss about a.i. if all it is is a prediction tool? the answer is because prediction is a key input to decision making, and decision making is everywhere. it is riddled throughout our business lives and our personal lives. and so this is the structure of the book. it's in five sections. in section one we talk about prediction, we explain what's so interesting about the new method of prediction compared to traditional prediction tools and statistical tech -- techniques. and while a.i. is new, decision theory is old. we have about 50 years of a well-developed decision theory. and so we could actually have a pretty good sense of what happens when we take this new prediction dispj put it inside a process of decision making that we understand quite well that's been studied for a long time. that's section two. you can think of them as the two academic parts of the book. then we get into the practical parts, section three is on tools

which is the actual building of a.i. tools. and in that section, i'm not to going to -- i guess i'll just give it a couple minutes on tools. the basic idea here is that we can take any work flow, so a work flow is inside an organization turning an input into an output. so you can think of a line of business as a work flow. and is we take the work flow and break it down into tasks, and every task is predicated on a decision or a couple of decisions. and a.i.s do tasks. they don't do work flows, they don't do jobs. they do it is a things that are predicated on a decision. so, for example, this is an article many people here would have read by the cfo of goldman. and in this article they open up with this dramatic statement by saying at its height in 2000, goldman employed 600 traders, today there are just 2 left. that's the dramatic opening, but later down in the article they have a more interesting sentence, so they talk about now

they're working on more complex areas of trading like currencies and credit. they're trying to emulate as closely as possible -- you can replace that with predict -- predict what a human trader would do. goldman has already mapped 146 steps taken in any ipo. okay, so the idea here is that they've taken the ipo work flow and broken it into 146 different tasks. and so what we do when we are building a.i.s is we take each task, and we estimate the return on investment, the roi, for building an a.i. to do that particular task. and then we -- the way to think about it is we just stack rank order the tasks, which ones have the highest roi for building an a.i.. right now this is not for the googles and facebooks who are already well into this, but, you know, the 99.9% of other organizations who are just getting into a.i., this is a approach we use to getting standard started.

we map out the tasks, we rank them, and then we start at the top and start working our way down. okay. and so sometimes there are companies that come to the creative destruction lab, large companies and say, hey, we've got three a.i. pilots at the company. are we at the frontier of a.i., and so just to calibrate, you know, google has almost 2,000 a.i. tools under development, and they're probably the high water mark. or possibly baidu. so we built this thing, it's in the book, it's called the a.i. canvas, and we have found this a very useful tool for getting companies started with a.i.s. and so what we often do is let's say there'll be 50 people in an off-site, and they come from all parts of the company. they're usually vice president level and above, and they sit in tables of four, and halfway through the day they go into these breakouts, and they fill out this page. and very often this is, there are -- the whole set of people, not a single person in the room

has ever written a line of code. so they don't have to have any technical background, but they simply go through their work flows, and they pick out tasks, and they say, okay, what if we built a prediction machine, and this is the prediction -- so they specify the prediction -- and they say with that prediction here's the human judgment that would be applied to the prediction. this is the action that the prediction motivates, and then here's the outcome that will result from that. and then the training data, the data we use to run the a.i. and the feedback so the a.i. can learn x. so people with no technical background fill out this thing, and by the end of the day we have, you know, an organization has a list of 20 a. i.s that are, have been just designed at a very conceptual level by senior people of the organization. and that's, we've found, a useful way to help them get started and just create a map of the organization of where they can start building a.i.s. okay. so the last thing i'm going to talk about is strategy. so most of the time when we

build a.i.s, we are billing tools that are very specific -- building tools that are very specific, and their role is to just make a process more efficient in the service of executing against the organization's strategy. so just like microsoft word or excel, an a.i. is a tool to make you more productive executing against the given strategy. but occasionally an a.i. can so fundamentally transform the economics of a process that they change the strategy itself. so this is what the popular press often refers to as disruption. and so what i'm going to show you now is the process that we use for helping to give us guidance on which a.i.s might lead to a disruption, a strategy change for the organization. our view is that the main thing when going new what i'm about to

show you is to develop a thesis on time. so in other words, the thinking about the time it will take for what i'm about to describe the happen will very much influence the investment decisions made today. in other words, whether something will happen in three years or ten years, it has very different implications for the investments you make today. so if i were giving this lecture even two years ago, most of what i'm about to say would be if. if this were to happen, then would this be interesting. or if that were to happen, then imagine the possibilities. over the last 24months, as most people already know already, we've had way too many proof of concepts to be thinking about this as an if anymore. in other words, we've had these in vision, so you can think of that as, basically, machines being able to see, giving them eyes. in natural language processing, so allowing them to comprehend words and language, effectively predict what those characters are trying to communicate. in motion control, robotics,

being able to do things. so this is no longer a discussion of if, it is a discussion of when. so we know it's possible, now it's all about turning and getting those predictions up to commercial-grade levels of accuracy. so we are in most cases now in the turning the crank mode as opposed to in the wondering if it's even feasible. so here's the thought experiment. we use a process we call science fictioning. but it's not science fictioning in the process that you can -- in the sense that you can sit behind a desk and think a.i.s can do anything. it's very specific. the thought experiment is imagine a radio knob, and you can turn the radio knob. but instead of turning up the volume, when you turn the knob, you're turning up the prediction accuracy of an a.i., so that's the only thing, the only parameter you're howed to move is turning -- you're allowed to move, is turning up the dial on the prediction accuracy. so here's the thought

experiment. everybody's been shopping on amazon, so we use this as an example that everybody will be able to imagine. and when you shop on amazon, you know, you are faced immediately introduced to an a.i., that's the recommendation engine at amazon where they recommend, oh, we think you might want to buy this, you might want to buy that. and right now for myself and my co-authors on average that recommendation engine is about 5% correct. meaning out of every 20 things it shows us, we end up buying one of them. that might sound lousy, but it's not too bad when you think there's millions of things in the amazon catalog, and it's going and pulling out 20 of them, showing them to us, and we're buying one. as you know, we go into amazon, the recommendation engine shows us some stuff and we shop around, and we see things we like, we put in our debt, we pay for it -- in our basket. and the order arrives on someone's tablet, the robots

bring it to the human, the human puts it in the box, puts a label on the box, ships it to your house, it arrives at your door, and you bring it inside, and that's how you shop at amazon. we can summarize that by calling that method shopping then shipping. you shop for the stuff, and then amazon ships it to you. okay. so here's now the thought experiment. imagine that recommendation engine, which most of us just breeze by, we don't put a lot of thought into it when we see it on the web site. imagine that, you know -- well, we don't have to imagine, this is just happening. every day at amazon the people in the machine-learning group are working on turning the knob. so let's say right now that knob is at a 2 out of 10, and they're working at improving their algorithms, they are testing some different approaches, they are acquiring data assets like buying whole foods so they can learn more about how you and i shop offrhine. each time -- offline.

each time they do that, cranking the knob. maybe they'll get up to a 3, then a 4, then a 5. they're entirely focused on turning the knob, increasing the prediction accuracy. as they increase the accuracy, they don't have to get up to spinal tap levels. maybe it's a 6 out of 10 or 7 out of 10. there's some number where when they reach that number -- let's say it's 6 out of 10 they get right -- somebody at amazon says, you know, we're so good at predicting what they want, why are we waiting for them to order it? let's just ship it. ...

>> where they may have told you to those things from their competitors either online or off-line then maybe only bought five of them may be the sixth you wanted but not really but now that it's right on your doorstep you might as well keep it. furthermore, in a fit for back on your porch that you did not want and it's in their self-interest to invest in the fleet of trucks to drive down your street to pick up the things they delivered that you didn't want. now i'm not sure amazon will ever do this or if they ever thought of it this is patent

that they filed with anticipatory shipping and they already started to test this and some markets were closed -- were just clothing items but the thought experiment is very powerful. so the only parameter was to turn in the so what is so interesting nothing happens nothing happens then the threshold or everything is changes the entire model shopping to shifting then shopping and shopping front shopping on the front porch and to integrate into a fleet of trucks so just imagine if you could predict insurance claims or bank loans or the

acceptance into the mba program there is some number where you crossed the threshold with a completely different approach when google announced they move from the first strategy to ai strategy it means something though mobile first they just want to be good at mobile that means they will be mobile at the expense of other stuff. in order to be mobile first. so what does that mean to be a.i. first? to put the dial at the top of your strategy it is your number one priority.

to trade off privacy to crank the dial. but peter is a research director that responded so with and for information anything with over 80% recall is good so the user can ignore bad suggestions. you would not use that service that but the wrong reservation 20% of the time or even 2% of the time. so this is much accurate so that is what we call a.i. first so from a computer science perspective but from the economic perspective add to that the trade-off when you make a.i. first everything is

second and third and fourth. and then moving people outside the ceos office with people working on a.i. so that brain team sat in a small office building but over the past months right where the ceo and other top executives work. so to conclude when people come we notice this business which is they arrive and say i get it. i get the amazon recommendation engine.

siri does not understand it here's the audio signal and predicts the response so people say i get it they are amazing but they are not transformational. but this is the chart of venture capital and the white house released four reports on the american economy and as far as we know the only technology over two quarters. then the google announcement moving to the a.i. first dreaded g than the government of china announces their strategy credible amount of resources to compete with a.i. by 2022 catch up and then to dominate and by 2030 across

every field. then in september the president of russia announced the a.i. of the future for all humankind and countries that meet a.i. then later on that month we hosted toronto the greatest gathering the economist the former treasury secretary larry summers and that chief economist of google and microsoft and with the economics of ai so at the end of the meeting the thinking fast and local food with this so a well-known novelist wrote

to me he is planning a novel about a love triangle between two humans in the robot so how is the robot different from the people? one is obvious the robot would be much better the second is it has a much higher emotional intelligence. isis will always be better and a.i. is already better to detect minor facial changes and then to detect one's voice they are getting happy or sad or angry or jealous? so wisdom is not having to narrow down a few. and that is broad framing of robot is endowed you wiser because we do not have a broad frame.

it is very easy to improve upon us i don't think there is very much that we can do the computers will not eventually learn to do so on the one hand and and how do we reconcile what we need is time but the dial of most applications is two out of ten and they are working 247 and moving faster in the last we had a technological revolution and

not just google's and facebook's also the older economy companies and then one company with less than a year so to sum up there is a nice quote that you not to long ago with the race between u.s. and china with the incredible amount of resources. and this is the sputnik moment and then to kickoff the space race with the creation of nasa

and what i think here is that it isn't just for defense but a sputnik moment for all of us. for all of us that are in the position to the organization but this is once in a generation. something like this comes along with the potential that this has. for the kids in the audience to a large extent you have the creativity how to apply the prediction machines in the way that the rest of us don't even think of. so in our view this creates an opportunity like most of us will never have again in our professional careers. so to have a set of opportunities there will be

challenges but the biggest is to make sure to have potential to change the structure socially. thank you very much. [applause] i can take some questions. >> my name is patrick. going to the one diagram that you presented on judgment and action, do you see a role for innovation in the prediction

process or is it something more constrained to judgment and action? so our machines best at doing that? >> so repeat the first part so repeat the first part innovation or creativity? i can see that judgment and action. and with that prediction? >> so i don't. i might be missing something in the sense that nature there is probability distributions that describe around us and simply to be better than we are to understand that probability. so that's not a matter of creativity but higher resolutio resolution.

>> i have really enjoyed this. hopefully i can read the book. so clearly fan of dna so you buy into the economics so that cuts with human beings are rational maybe they are not. but i want to hear your opinion to make strong assumptions if you have enough data or if it has a correlation and then a magical black box. so we are in the rational and then it will work out with the

classical economics and what were your thoughts around me be with this magical vision data so what is this analogous behavior? or to be over exuberant? >> sure. so to be over exuberant in a sense that first of all that they can do things other than prediction and it can only

really do those predictions where there is good data. so then we don't have good data and a.i. cannot make good predictions but i would say where they feel magical because they can find patterns in data that traditionally that we did not have access to. with image data with the lens of data with the excel spreadsheet -- spreadsheet that can complement the process to make that prediction and furthermore, before we started to make a prediction we created a model a boyd used to predict the outcome. now to use the kitchen sink

approach and let the machine figure out to make that prediction. on the one hand we have a much more predictive ability than people realize because the prediction machines work but there isn't a silver bullet and in the first section with all the limitations because they don't have model and then rely on the underlying data. >> i am happy to stick around for off-line questions afterwards. >> it is clear for me that you can establish monopolies but there is absolutely no reason

why those are uniquely retained by human beings. so in and to become more valuable. a lot of people say with human beings. so what reason do you have really to feel that's true? >> that is a great question. i will attack it with just one example. in some sense that is received to take the most refuge. and the question was largely are we still in the first inning and this search moves

to this part? at least in the moment with the prediction that is all that is. just prediction. and with those type of judgment and then to break in that prediction and that the a.i. is doing our judgment step on the gas or the break the pending if it is raining or how fast we are going in those judgments are baked into the decision. and then that is what everyone to do but i think of this to

some extent that once a.i. is good at predicting judgment now we can focus on a new set of problems but at some point you run out. but one thing we are very poor out -- with the monkey brains and so just the same way with those of the population. and where 45% are no longer in agriculture. nobody raised their hand to

say they will be game developers. nobody has done that. nobody even imagine such a thing would exist. so we are poor at imagining with high prediction machines what are all the other things we could be doing? most people would say our healthcare system is not bad but years from now we will say we cannot believe they were living in those horrible conditions back in 2018 because they will do so much more efficiently and intelligently but they all sorts of things and then we are still scratching the surface.

>> and then to look at is there an option or are there some ways and what to go after? but that is the process to estimate to build the a.i. how long will it take or how much will it cost? and with that performance. so to describe that process. then to help people get started just far enough for non-technical people to provide more definitive answers like how much data we

need or what types of sensory information we need to collect to do that. but a reasonable step-by-step process of where to start and what to focus on first. thank you very much 17. [applause] ♪

>> when the president immediately accepted the offer to meet to north korea my first reaction was oh my god what is he is doing and then i thought nothing else has worked so maybe they do have an opening. so i would just say the following. the first thing is we do know there is a north korean

pattern and when you try to negotiate with the father. we know there is a north korean pattern they get in trouble and get isolated and sanction start and then they come to the table. and then they don't live up to them that they do up to a number of promises like dismantling pyongyang or the cooling tower but then you learned they had a hidden uranium program they will not admit to it so then you have to end the negotiations. so a couple of things that look different this time the leader is different but because north korea was getting close to a capability to reach the territory of the united states that people began to take the american

president or seriously when he said that's not acceptable i know from the alliance management standpoint it was when the threat is regional but now threatening california or austin and then that the united states could actually go to war. second we had a change with secretary of state but let's give rex tillerson credit for the isolation campaign he organized against north korea. with the expulsion of north korean workers from 20 countries the regime was also starting to run out of spare parts militarily and by the way some of those luxury goods it was on brandy and cigars because that is what the regime wanted.

so now they set the table in the effective way. so the first is others have equities here like the japanese. be careful not to go around. and take your time. with the removal of the american military forces on the korean peninsula are stabilizing for the region as a whole. so be careful of the structure. but do not forget the nature of who you are dealing with. this is a regime less than a year ago where the leader killed his half-brother under

chinese protection in malaysia using gas. they have death camps for their own people. so never forget who you're actually dealing with. if you can get inspectors on the ground do it the intelligence is not good so inspectors on the ground can matter that take your time. don't try to negotiate at the table with kim jung-un. let the experts do that.

>> my name is jessica i am staff at young american studies and i would welcome you all. welcome to