one. ♪ we are ready to get started if people are out having breakfast could come in to join us that would be great we have folks coming up from the entryway downstairs to make sure they are filtering in.

i realize this is a little bit of an oxymoron for the timing today but they keep her coming. i have a partner here and i have spent much of the last five years trying to figure out the economics of artificial intelligence i have to confess i am not yet there i have to work on it but thanks to our speaker today who has spent a lot of time on this subject and has published papers and has written a book that he will talk about today. my had the opportunity to work with him on multiple projects and what is particularly interesting to distinguish an academic and as a successful serial entrepreneur in artificial intelligence on the academic side of professor of entrepreneurship at the

university of toronto focusing on artificial intelligence and also a research associate at the national bureau of economic research in boston and i'm curious he has founded three entities all associated with artificial intelligence that have been interesting one is about to open operations in new york but the first is called the creative destruction lab as an advisory organization starting in toronto it is an accelerator home to 150 ai driven startups that is the largest concentration on the planet. so they will be extending operations into new york now a program for young ai entrepreneurs in canada and the third is kindred which is a startup to build machine

with humanlike intelligence recently named through the tech review smartest companies of ai 100 also in 2017 national committee newspaper named him one of the most 100 powerful people in canadian business and today he will talk about his book prediction machines. published worldwide this week by harvard business review press. he will speak then we will have 15 minutes of q&a and then we will have coffee and discussion outside afterwards and as you saw there is a big stack of books and take went on your way out with that as a

background please help me to welcome our guests. [applause] >> thank you very much for hosting and i don't know if kelly is here but she coordinated mileage is six. and also thank you to the literary agents who helps to bring this book to fruition just so i can calibrate how many people have an ai company? so in terms of familiarity if you had to stand up who would feel comfortable in a few sentences defining what ai is?

three or four? and those organizations to characterize where you would place your organization with that strategy. and to have that strategy category one or category two to put together a team to work on that and category three is we have no idea where to start those in category one? three people. category two? medium category. category three? so hopefully by the end of this for the first question

everybody feels comfortable you could define that that you have a sense of where to start. i don't need to tell this audience that you self-selected to be here but and what economists call general purpose technology seems to be everywhere it is hard to find any market where there is not some ai applications rolling in to the advanced productivity of what they are doing but at the same time there is an how they deploy that and what that means for humans.

so what we try to publish with the book is remove this anxiety by way of providing clarity how do we think of those without a computer science background and we take a topic that is the domain of computer science to put on a different lens from discipline we don't associate with clarity or economics so on top of the computer science background in this case we can have a fair amount of insight into how we think of development of ai. as an economics professor at university of toronto this matters because two blocks down from my building is a computer science building.

due to serendipity, toronto was the epicenter of the recent renaissance. so today the avril industrial groups are headed by those by toronto. those that originally led the ai group here in new york on facebook was at university of toronto. one -- ai research for elon musk ten years ago university of toronto. the reason that's relevant is

i am the founder of a program that is opening here under the leadership called the creative destruction lab to transition science projects into massively scalable companies. so five years ago what started as a trickle turning into a flood was graduate students in the companies coming from around the world but either this particular lab the first one that came that said i will use this new technique to predict which molecules that will bind with which proteins and then right after him

another one came so to predict which credit card transactions are fraudulent vs. legitimate that another one that said to look at medical images to predict which tumors are benign versus malignant. or which automobiles have defects before they roll off the production of mine. so were sitting at the lab it didn't take a rocket scientist to figure out there is something unusual coming out of this computer science department this same technique of deep learning to have a list of wide set of applications after a while we

started to document what we were seeing. what are the general lessons to be learned that are applicable across these different cases? that resulted in this book. so what i will do now is give you the key points. so i don't need to say to this audience a lot of people feel that today in ai feels a lot like 1995 felt with the internet. most people will remember that 1985 was a real transition year for the internet we had them for a little couple of decades with the military and

academics and it was growing over time then there was a big jump that bill gates wrote microsoft windows add windows 95 and in august of that year three billion-dollar ipo for a company that generated almost no profit. so the language around the internet started to change and we start referring to the internet is a new technology and as a new economy but the point was it permeated so many parts of our lives that we stopped thinking as a technology but a different way to interact so ceos and

journalists and entrepreneurs and investors and politicians refer to this as a new economy. except one group of people that was the economist. and they said wait a minute this is the exact same economy we have always had in fact will have to change a single word or page all those economic models still hold everything is driven by supply and demand it is all the same. the only thing that has changed is the relative cost of key inputs have fallen dramatically to distribute goods and services cost of search and communication and

that's the way economists view the world. what they are quite good at is to take that you technology to strip the fun out of it and resolve down to a single question what does this reduce the cost of? so that gives some great insight the heart of silicon valley is a hard as summer conductor industry. so in their heads they have an image like this to describe to you the underlying science to cram more transistors onto a chip and explain that chip doubles every 18 months that is what they will explain to

you but to ask the same question to any economist the rise of the conductor industry they will have this image. and the reason they say they are so foundational while there are many things happening in the bay they are the engine is because this thing of which the cost fell was the input so economists think of semi conductors and how many people saw the film hidden figures? does anybody remember the job title of the women as the protagonist? peters one -- computers because they came to work and

they computed. there is a scene with a roll the computer people try to figure out what that means for them. so three important things happen. understanding these three things to figure how this will affect the economy. going back to economics 101. with that downward sloping command curve. and so for example those that of arithmetic and then we started to use a lot more. better faster cheaper we

started to do more of that. also where things become interesting is we use more of it not just arithmetic problems but things that weren't and convert them to take advantage of the new chief arithmetic and photography's to be a chemistry problem we saw that with chemistry to make film. but as it became cheap we change the arithmetic base of communication and banking one thing after another to transition to that solution. now to ai. if we were to ask a technologist please describe the rise of ai they would

describe the rise of the science and the statistics to talk about inputs and outputs and their weights and they would explain the development of the process. but instead you ask any economist can you please describe the rise of ai they would not have this image in their head. some people think wait a minute they are single-minded but at some level that's true but this is the key to understanding why economists think of ai as a category of its own so go to the consumer electronics show in las vegas with a rainbow of new technologies so you see so many different areas of tech

there are so many things why is it so special? to say it is in a different category. and that is a foundational input into such a wide range of activities and that means prediction. we can think of the rise of ai is a drop of cost prediction. and that will replace that with the chief prediction. and that will seem less magical. so first of all how to redefine prediction to take information you don't have so that concludes like demand

forecasting like the last five years of sales to predict a forecast of next quarter that is an obvious former prediction but less obvious like classification looking at a medical image with those pixels in the information we don't have is if the tumor we king at his benign or malignant so we call that prediction. it takes the information you have to generate what you don't have so we have a drop plummeting and the cost of prediction what does that mean or the applications for business? and society? implication number one is now we have the demand curve when the cost of something falls is more of it.

first of all all the things currently like supply chain management and insurance it will simply use the new superpowerful prediction so that will replace traditional testicle technique. at the same time where in my view on the business side becomes an arch more than a science that people start converting their prediction problems into another. and for that example there is driving we have had autonomous vehicles for him.

but we bullies deployed our autonomous vehicles in a controlled setting in a factory or warehouse in the way we did it is that an engineer would have the floor plan and program a robot to move around the factory floor. then they give the robot a bit of intelligence and put a camera on the front until the robot if somebody walks in front, then stop or to the next shelf. if that.

but there is an infinite number of uncontrollable world. because there is an infinite number of its ago they said the experts said we cannot have an autonomous car in our lifetime because we cannot program all of the gifts until people in the machine learning field be framed the bubble to say rather than the infinite number of change the problem of one prediction what would the human driver do? so imagine a putting a human in the drivers seat imagine ai

in the passenger seat. so we told the human drive. visits in the car and starts to drive then they have data coming in through cameras and microphones then they process the data with the monkey brain and taken action that is very simple turn left, turn right, break and accelerate. many if but a small number of events. so imagine it is sitting inside the human it doesn't have his own eyes or ears so imagine the comes in as you are driving and every fraction of a second ai is looking over

to predict what will the human driver do in the next second? in the beginning it's not a good predictor. they're not very accurate. they said think she will turn left or go straight or break but then something happens. she does or she doesn't turn left every time they make a prediction if they are right they double down if they are wrong they update and then make a different prediction the next time. so as they are driving they make a lot of mistakes but as they drive and learn incorrect then it gets smaller and smaller until it some point it is such a good predictor of what the human driver would do that we say it can do it

itself. so it is a prediction machine for driving. that is where the enthusiasm is because it is converting problems that were not prediction problems into prediction problems. driving is obvious but we have done it like translation that used to be rules based. linguists who are experts in the rules of translation and they would do translation now we have converted that into a prediction problems so now google translate even between one year ago and today the improvement is significant and it's not too far away we'll probably have a commercial grade translator based on prediction rather than rules. at the creative destruction

lab this is where we have all of these ai companies and as far as we know home to the greatest concentration they are coming in from all over the place and these companies each one is working on solving a prediction problem. and those that better understand ai an interesting group so that conversation we have that they will fly to toronto to learn more about ai but i need to know for recruiting what type of skills. in order to prepare them for ai we need to know for the other parts of our company the sales department but not from

my department hr is very human but for the other parts of the company we need to learn about ai for recruiting. most of you know where this is going. one by one ai companies are transforming the hr process into a series of predictions. what do hr people do? first they hire and recruit recruiting is a prediction problem with a series of resumes and cover letters and interview transcripts then predict from the set of applicants what are best for the job. once we hire people we do promotion what is that? a prediction problem set of people and we have to predict who would be best. then the next issue is retention with the 10000 person organization to retain

our best people it is a prediction problem we need to predict which of the stars would be for about keeping them so one by one these roles are converted into prediction problems. so item number one so we use more of the cost for traditional things and also convert new things into prediction problems to take advantage of the better cheaper faster prediction. here is number two and number three. when the cost of something falls, it affects the value of other stuff and economic language basically you can think of things being related to the focal thing as comments or substitutes think of coffee

if the cost of coffee were to fall, what we use with coffee like cream and sugar it would go up then we consume more coffee than we also consume more cream and sugar therefore the value will go up for the cost of golf clubs falls the balls go up those are complements the value of complements go up but substitutes go down so if coffee falls then some people switch from tee to coffee then tea falls because now the demand diminishes so now how does that work with ai? we can take any task and break

it down into these components. any task it doesn't matter what port of the organization you work in there is a series of tasks and everyone is broken down so as i am giving this talk right now three days from now my knee is sore so i go to the doctor i tell him my knee is sore so she says mom -- he asked me a question and sends before the expiration makes a prediction that might be with 90% probability you bruised your knee 10% there is a hairline fracture. then she applies judgment and that is how costly would it be for this patient if they

actually have a bruise but i mistakenly treated as a fracture versus if they actually have a fracture and mistakenly treated as a bruise? that is her judgment taking all the things that she knows about me to figure out the cost of the mistake. that is judgment but then she takes an action. so it is a bruise put some i.c.e. on it and come back to see me in a week if it still hurts. the action than the outcome and then we have learned she was right and then we have feedback data to strengthen the model or one week later my leg is worse than that is feedback data we update and change the model for the next time. any task we can break into these components and this is

very useful for designing strategy and those implications for jobs in the economy and here is how. if you look at this diagram it is very clear what is the substitute for machine intelligence? as the cost of machine prediction falls it is the human prediction the value of human prediction will fall as the capability of machine prediction increases. we are poor predictors we are slow and noisy with systematic biases that is well-documented and books like thinking fast and slow predictably irrational these all document how terrible humans are making

predictions that we still make them all the time. so as the capabilities of machine prediction increase the value of human prediction will plunge. so that becomes less and less valuable because they can do it better faster cheaper so that is the part the press is fascinated with leading to a mischaracterization of a complete wiping out the role of humans. what they miss are all the other boxes those are the complements like the cream and sugar that will increase in value as machine prediction falls. talking about input other people have heard the phrase data is the new oil? that's talking about the input

we have always had that data for a long time. why is that the new oil? what makes it new it is way more valuable today than it was ten years ago because the cost of prediction has fallen so the value goes up it is a compliment because prediction is better faster cheaper so the press has done a good job talking about the input box but it has not done a good job to describe the other of human judgment. they don't do judgment we have to give guidance. so for example the doctor who decides the cost she is using

her judgment. so what is interesting really supplied the judgment but the value goes up as the cost of prediction goes down as we start to have better faster cheaper predictions in the fidelity increases the value goes up to a much better prediction. so predictions are used to form an action what action should be take? operating companies that is a valuable asset not just data is the new oil but the actions are valuable because you can take better actions on higher fidelity prediction so in our

lab with these ai companies and issue is they build and sell predictions but they don't own the action so they have to sell the predictions to somebody who does because without that they are worthless. actions are a complements the value goes up where we have the most amount of negotiations with those ai startups is the feedback and that is the gold dust when we take the action we find out later it was good or bad and that is how we learn to update the ai so all of those startups try to license their prediction to sell their predictions are all trying to get their hands on the feedback data now the company realized that is the gold dust much more valuable than this

data. if you use the oil analogy you burn it and it is gone you use it to train the first time once you train broadly then it's done and the value is gone. it comes from the feedback data that allows them to continue to learn. so the key points in thinking about strategy and the implications of jobs the value of prediction falls but the value of complementary assets go up. a useful way to think about this example is think about when spreadsheets rolled into town and accountants originally think of them as a skills test wind was to type in the numbers and add the

value their ability to type and add fast but then second was to ask good questions to estimate the present value of assets then they may ask a good question what happens if interest rates go up by 1% or sales went up 4% in the first quarter? but as soon as they ask they have to retype everything and re- at all the numbers to test the new scenario. spreadsheets rolled into town the value to type fast and add fast went down because the machine could do that but the value to ask good questions went up because it was quicker and easier to ask a question if you are good at asking questions with the scenario analysis your value went up

your key skill was adding faster value went down. if ai is just prediction the current renaissance is about lowering your cost of predictio prediction. then why is there so much fuss about ai if it is just a prediction tool? the answer is because prediction is a key input to decision-making and that is everywhere it is riddled throughout our business life and personal life that is the structure of the book in five sections we talk about prediction and explain what is so interesting the new method compared to statistical techniques and then decision-making. ai is new but the decision theory is old we have 50 years of a well-developed decision

theory so we could have a pretty good sense of what happens with the new prediction technology employee inside the process of decision-making we studied for a long time that is section number two. theory is old we. . have 50 years of a well-developed many people here would have read

and in this article be opened witthey openwith a dramatic stay saying the equity stream employed 600 traders and today there are just too left. that is a dramatic opening bit later in the article they have a more interesting something they talk about working on complex areas of trading to emulate as closely as possible you can replace that with predict. goldman has nabbed 146 steps taken so the idea is they've taken the workflow and broken into 146th and tasks and so what we do when we are building is we take each task and estimate the return on investment and then the way to think about it as we stack the rank order we start at

the top and work our way down. the 99.9% of other organizations are just getting into ai this is an approach we used to get started. we bring them and then we start at the top and start working our way down. sometimes there are companies that come to the lab and say we have three pilots at the company are we at the frontier and adjust to calibrate a there are 2,000 tools under development and probably the high watermark or possibly w they built this thing in the book and we found this a very useful tool for getting companies started so what we often do is let's say

there will be 50 people in an off-site and they come from all parts of the company vice president level and above interested in tables of horror and halfway through, they fill out this page. very often not a single person in the room has written a wide code that they go through the workflow and pick out tasks and say what if they built a prediction machine and they specify and say here's the human judgment that would be applied, this is the action that the prediction motivates and the outcome. then the data that we used to run the feedback. so people with no technical backgrounbackground fill this od bite the end of the day the organization has a list of 20 that have been defined at a

level by people o with the organization and we found a useful way to help them get started and where they can start building ai. the last thing i'm going to talk about is strategy. most of the time we are building tools that are specific and the role is to make the process more efficient against the strategy. occasionally it can transform the economic progress and change the strategy itself this is what it's referred to as disruption.

i'm going to show you now is the process that we use for helping to give guided that might lead to a disruption to change for the organization. our view is a. thinking about the time it will take for what i'm about to describe will happen will very much influence the investment decision made today in other words if something will happen in three or ten years is different from the investments made today. if i were giving this lecture even two years ago boasted what i'm about to say would be if this were to have him would be interested or if that were to happen imagine the possibilities. as most people probably know already, we have had way too many proof of concepts to think

about this anymore. so you can think of as being ablabout is beingable to see gin a natural language process to comprehend words and language effectively to predict what those characters are trying to communicate this is no longer a discussion of if but it is a discussion of when so w we knows possible and now it's at getting them up to commercial grade levels of accuracy so now we are wondering if it is even feasible. here is a thought experiment. we use th a process we call scie fiction but it's not i not an oe if you can sort of sits behind a desk it's very specific. the thought experiment is imagine a radio knob but instead

of turning up the volume you are turning up the ai. we use this as an example everybody will be able to imagine. you are faced with recommendation agent and right now for myself and co-authors gleaned out of every 20 things at the shows we end up buying one of them. it is going out and showing them to us and we are buying one.

the. they arrive on the tablet and the fulfillment center and a figuring this stuff up, the human puts it in a box, puts it in a label should stick to your house and it arrives at your door and you bring it inside and that is how we shop on amazon. we summarize that by calling that method shopping and shipping. okay. so here is now the thought experiment. imagine that recommendation that most of us just breeze by we don't give a thought into it when we see it on the website. imagine that -- every day people

are working on turning the knob selects the right now it is two out of ten if they are working to improve their algorithms into testing some different approaches. they are acquiring data assets so they can learn more about how you and i shop off-line. maybe it only gets three, four or five. they are working on increasing the prediction accuracy. as they increase the accuracy they don't have to get up to the spinal tap level levels maybe st of ten or seven out of ten. there is some number where when they reach that number let's say six out of ten they get it right somebody on amazon says we are so good at predicting what they want why are we waiting for them to order it with just ship it to. so go through the thought experiment you get up to the

level they say let's not wait we any more let's just ship it. the box arrives and you take out whawhat he wants to let's say sx out of the ten and you leave four of them. why would they ship things they know you are not going to take six they are significantly increasing their share so whereas they might have sold you two of those things and you might have bought four of them from the competitors online or off-line now they are selling you all of this. but it's right on the doorstep i might as well keep it. furthermore now you've put things back if you didn't want and it's in the resultant wrist to invest in a fleet of trucks that are going to drive down the street and pick up all the things they delivered to yo deld

your neighbors that you didn't want to lower the cost of handling the return to. i'm not sure amazon will ever do this. it's not like they never thought about it. they've already started testing it in some markets with clothing item in some markets, but the point is that the thought experiment is very powerful and remember the only parameter that we moved its turning the knob. what's interesting about turning the knob is as you turn the knob it's like nothing happens, but we don't even really notice. nothing happens and then it hits a threshold where all of a sudden everything changes. it's a shopping and shipping to

shipping then shopping and a vertically integrated into a fleet of trucks and so on so you can imagine and predict insurance claims or bank loans or acceptance into programs as a member you cross the threshold for a different approach. okay so when google announced last year they were moving to the first strategy is that just marketing or does it mean something? it means something. when they said they were mobile first it wasn't just that they want to be good mobile first means that they will be at the expense of other things. they will sacrifice the website and physical stores or whatever in order to be mobile first.

that is their number one priority. so this means putting the dialect the top of the strategy priority is. you will trade off other things, the user experience and revenues and privacy to crank the dial. the research director responded with information retrieval and anything over 80% recall position is pretty good. not every suggestion has to be perfect since they can ignore bad suggestions. with assistance, there is a much higher barrier. you wouldn't use a service that booked wrong subsystem needs to

be more accurate and intelligent and aware of the situation. that's what we call a ai first. we would add to that the trade-off when you make at first you make other things second and third and fourth so they put this as a priority turning the knob. one example of making this a priority is reshuffling the deck moving people who are outside of the office somewhere else in moving the people working on ai in the same office as the ceo. so they sat in a small office building on the other side of the campus but the last few months if switch to buildings and is now with the ceo and other top executives. to conclude, when people come to the lab we notice they arrive

and say i get the recommendation and prediction. it doesn't understand what you say. here's an audio signal and predicts the words you're saying and predict the response for what you said. so people say i get it these are clever and amusing but they are not transformational. on the other hand this is a very steep curve the white house released records on how to prepare the american economy for what was coming around the corner. as far as we know and we can find it's the onl the only teche white house has released the reports since the second world war. then there was the announcement followed by a series of other companies moving to this strategy.

thethey must july the governmenf china announce announced their y with athis strategywith an incrf resource to compete and with the goal of by 2020 catching up in some fields by 2025 dominating in a few subfields and by 2030 dominating across every field with the aspiration. then in september president of russia announcing it is the future not just for russia that all of humankind into the country that leads will rule the world. then later on in that one, we hosted what i think it's still the greatest gathering of economists to meet on the economics of the former treasury secretary and former chair of the council of economic advisers, chief economist at some o,the nobel laureate and os gathered to talk about the economics because computer science have gotten so far ahead

of economics and the subject. at the end of the meeting, the author of thinking fast and slow convoluted with this he said i want to end on a story a well-known novelist wrote he is planning a novel about a triangle between to humans that are robots and he wanted to know how it would be different than the people and i proposed three main differences that would be better for reasoning. second have a higher emotional intelligence, so we think of us as being always better than machines and emotional intelligence that the but they t so. they are already better some domain of protecting minor facial changes to detect changes in mood and audio to detect when someone's voice is reflecting that they are happy or sad or angry or jealous. third is that the robots would be wiser.

it's not having a narrow view that is the essence. broad framing. it will be endowed when it's learned enough and wiser we are narrow thinkers and it's easy to improve upon us i don't think there is very much that we can do that computers will not eventually learned to do. on the one hand, we have these things that we see that looks neat but they are not transformational and on the other hand we have all these people who are making claims implying that it will have this effect on the economy. how do we reconcile these two things and in our view reconciles these things is time. having a thesis on time that the dial is sitting there at two out

of ten and people are working on cranking the dial to three or four and it's moving faster in some domains and slower in others. it's not just the last time we had a technical revolution. some companies have a good thesis on time and others underestimated how fast things would move. in this revolution isn't just google and facebook and apple come it is older companies making bets as everybody knows. we had one company that had virtually no revenues and it was acquired for $100 million. and so, there was a quote that came out not that long ag long e former secretary of defense referring to the race between the u.s. and china and the incredible amount of resources they are putting behind it.

she uses the phrase this as a sputnik moment of course referring to the soviets launch of the satellite and they kicked off the creation of nasa and so on. what i like here is that it's not just a moment for defense, it is a moment for all of us. in the position of the leading or running organizations, we get ones of these in a generation that comes along and has it had of potential that this has for the kids in the audience i think a large extent you will be the ones that have the creativity of how to apply the production machines in ways that the rest of us don't even think of.

the main point is this creates an opportunity like most of us will never have again in our professional career and for some of us, that will pose a set of opportunities in to seize and pursue the same time of course there will be challenges and to make sure as a society everybody benefits because it has the potential to change in the structure socially. thanks very much. [applause] i'm happy to take some questions. yes please. there is a microphone coming because they are recording it. if you don't mind introducing yourself.

>> my question is if we go back to the diagram that you presented on the prediction, judgment and action do you see a role for innovation or is it something more constrained and if there is a role for production or judgment and production will machines be best at doing that or is that a human complement to prediction? >> if you think of it is as innovation or creativity i can see the role for creativity and judgment and action you see in the long run the role for creativity and prediction. >> i don't thought i might be missing something in the same about in the world in nature there are a probability of distributions that describe the phenomena around us and it will

simply be better than we are at understanding what those are so that isn't a matter of creativity is getting higher resolution >> hopefully i will get to read the book soon. you are a fan and i assume you buy into the behavioral economics, so that kind of comes with human beings are rational maybe they are not. right now we make strong assumptions like do we have enough data and particularly if it is a vision problem or has correlation you are going to

have a precision and magical black box. i just -- my sense is that we are in the rational world like the assumption is we have these in enough data and it will work out and to me it seems a lot like the classical economics like this is just going to workout and i wondereworkout ane your thoughts around maybe they only work a certain way because we have a magical vision that it happens to correlate spatially and it works properly. what are your thoughts around if that is true it would be that analogous behavior of economics and are we exuberant about the possibility. >> there is room for being over exuberant in the sense that

first of all people imagining they can do some things other than production. where do you sometimes feel magical is because they are able to find patterns in the data that traditionally we either couldn't use or didn't have access to so we could censor us sconjure upso much stuff come in other words not just in a spreadsheet we used to use for making predictions we could use so much more data of all types that can complement the process

of making a prediction and furthermore in the old method we used to have t two before we started making predictions, we had to create a model in our head of what the hell we were going to use to predict what the outcome. now we can take a kitchen sink approach where a given everything we let the machine figure out what is related to what to make a prediction. on the one hand, we probably have much more predict that capability than people realize because of the way the prediction machines work on the other hand, there's certainly not a silver bullet. we have a whole section on what they are poor at because they don't have a model that we are basically statistical correlations that rely on the underlying data. two more questions and then i know people have to get going and i will stick around for questions afterwards.

>> you labeled the entrance and it's clear to me how some of those complementary boxes you could establish monopolies around but it seems to be absolutely no reason why they would be maintained by human beings. every single one of those is subject to research generally. and as they become more valuable you could speak to the research effort so a lot of people say that it would be great if they become more available. but what reason really do you have to expect that it's true? >> that is a great question and i will attack with just one.

that is where we seem to take the most refuge and okay we are safe from machines because we have judgment and they don't so the question was are we still in the first inning and research is moving into these other boxes. so, at least in that moment as long as the prediction that's all it is is just a prediction but the point is if they see him as examples of that particular type of judgment they can learn to predict that judgment. so it is effectively banking in on the judgment so when we are approaching a yellow light, it is learning and inferring our judgment on whether we will step on the gas or on the greek depending whether it is raining and how fast we are going. and it is being baked into their decision.

>> so it is true that things today that we called judgment they will get enough examples of making those judgments in order to predict them. to some extent you will like those, like the russian nesting dolls that once they are good at predicting a certain type of judgment, we now can focus on a new set of problems where the judgment is useful in the domain. the question is when you run out of them and there's nothing left. your guess is as good as mine. what we are very poor at, that we are not good at is anticipating things we've ever seen. so just the same way we would have asked people to hundred years ago where 40% of the population or 100 years ago the

population was working on agriculture if we said imagine a world where 45% are no longer in agriculture, it is only like 2% of the other 45% going to do nobody would have raised their hand and said they would be at a company with gaming developers, nobody would have done that because nobody would have imagined such a thing would exist. and that is what we are very poor at imagining when we have these machines but are all the other things we could be doing most people would say our healthcare system still isn't great, but it's not bad. i can't believe they are living with that system they had back in 2018 because we would be able to do so much more efficiently and intelligently because they are able to handle the same with space exploration where we got

clear of the frontier and in fact we are just scratching the surface. one more question and then we will wrap up. >> we have dabbled a little bit into solving different problems but there are a lot of different constraints. is the assumption that no matter the problem we will be able to return the file or are there ways in which we can make better decisions on what' what not to o after? stomach that is the process soon how long will it take, how much would it cost and what do we need to turn the dial far enough for this to give us a lift in performance. so we've described some processes on how to do that.

i think it is in front of the book to help people get started, far enough for the non-technical people to get started to know when to be able to bring in the technical people to provide more definitive answers in terms of how much will we need to do this or what type of sensory information we need to collect to do that. so there is no single answer to the question that there is a reasonable step-by-step process for where to start and what to focus on first. [applause]