survival turns into a problem the same problem is true. they are incredible. but this technology is amazing the agility is incredible. tantamount to the industrialization of food to do survive every day. but just like that in the machine it has the potential to short circuit going right to the nerve center with primitive social rewards that can hijack as. i hope we become critical of this the way we have of food. the metaphor that has been used before that i talk about is the book find a direct.

but it does take a concerted effort because we're just at the beginning what is fact or sugar or salt. >> host: we're out of time but you have left us with so much to think about and to be skeptical about the end we can think about the issue of new ways. thank you very much for writing the book. >> guest: it was a real pleasure. >> that was after woods booktv signature program were authors of nonfiction books are interviewed by a journalist and public policy makers and others familiar with their material it airs every weekend 10:00 p.m. and 12:00 a.m. on monday. you can also watch afterwards online go to

booktv.org and click on the upper right side of the page.&j >> nick bostrom talks about a future where machines are more intelligent than humans that it will try to save for destroy us. this one hour 45 minute program is next on booktv. >> can you hear me? yes? in the back? excellent. height and the exec director of the machine intelligence research institutes and we are of a nonprofit research institute two blocks off the uc-berkeley campus and very excited to be hosting the talks by professor nick bostrom the director of the future of humanity institute at oxford university we

covered wrigley and research related to this subject of machines super intelligence and specifically the challenge to get good#é&m outcomes from machines setter clever and more powerful than we are. have a couple of comments on logistics'. after the talk you can buy it of copy and ethnic will find some time to sign some of those. second, if you would like to learn more common miri is running an on-line virtual greeting group for this book we will redress actioníñ together for one week and discuss it in an internet forum on line. that may be the best way to learn more about this because you can read at the same pace and discuss section by section. just go to our web site

intelligence.org there is a link. third if you are researcher of computer science please do visit the miri research page to see if the problems we're working on is interesting to you and get in contact with us we'll always look for people who want to research the technical challenges when you figure out how to control the system more clever than you are. now i'd like to introduce stewart russell who received his doctorate in computer science from stanford in 1984. 86. of surrey and a faculty member ever since and won numerous awards and published dozens of papers and the co-author of the beating a i textbook in the

world. for many years has been asking the question what if we succeed?jr i agree this is an important question and recently organizing panels and workshops with the other day i scientist. it is an honor to have you here. [applause] >> double-a-2 thank miri for organizing this visit and also nick bostrom to berkeley as the professor at my old university at oxford when he directs the future in a few minutes. [laughter] he has an eclectic background to study physics

and ince and philosophy of science trying to find out where the big questions were. as a result he has become one of the most promw á philosophers not just asking the big3 question but has provided as much as anyone ways to think about these big questions leading us to change the way to change the human race. and our real special to us because we ask the question to affect our understanding.

do we live-in a simulation and? the answer is yes. [laughter] what happened to the civilizations that must be outdated? because they developed artificial intelligence? that is what he will address today so with that i turn that over tunic to save my conversations have convinced me much more than before as the essential task of our age. [applause] >>w thanks for coming. i am not going to try to

summarize the entire book but give some background like where is this coming from from a writer's perspective a and to end the presentation may be with you and dave we can tell into more concrete topics you're interested in. so to take a step back from the abstract point of view. with a grasp on the x axis or the wide access of capacity and advancement for perot is the normal way for things to be. if you think of it is a very abnormal way for things to be. the human species is young

even on the time school this condition every waking up in the morning you go to work and sit in front of a screen all day then worry about eating too much. but we have escaped to the condition and even in parts of the world of

downward direction in population with the concept of the species but even though it is very different but not immune to going extinct. but once they are they tend to stay that way. [laughter] but there is another attractor as well if we succeed to break out in the oppositekw direction, if we reached technological maturity to enabled the colonization of space the representative decatur's and in that direction that this

civilization and continues to expand and some fraction of the speed of light indefinitely and tell us some point the acquisition becomes impossible because of the expansion. there is a finite to bubble if someone is starving that they could access and then to break out that we gradually approach that. the risk of suffering the existential catastrophe but the concept of the existential risk is originating for the permit and drastic destruction about a potential for development. sova way we can prematurely and the human story.

the way we can permanently destroy the futurer' all the things that cannot go wrong but to be the existential catastrophe and it is only zero or one of these but it is important if it has to be of value were the value of a local entity fundamentally does not depend on space and time it doesn't depend on where it takes place whether here or in glendora africa fundamentally. if i have that you then it becomes a lens to focus our attention and what seems to

matter. you can see more in the future then-president with billions of galaxies their planets could have had millions of people for billions of years but if you multiplied the numbers together even a very small change of probability 100 off one percentageq points it is a higher expected utility they and anything else to eliminate cancer to eliminate world ponder. it maybe whatever in direct effect it has.

it is not necessarily correct but maybe a component so i would suggest it is this portion of it -- portion a part of character doing a survey a couple of years ago doing research on the dung beetle in human extinction we tried to change that to save the world one piece of paper at a time. [laughter] but the one finding of the big existential risks will

rise from human activity. there are risks the we can see they could be small we have survivedee earthquakes and asteroids so it is unlikely it will happen in the next hundred but particularly for of future technologies to expand the powers from the external world and ourselves. said different way to make the same point is the metaphor of the big urban -- the urn of discoveries and technologies landed. throughout human history we made a bunch of discoveries.

the net effect is extraordinarily beneficial that now lives in abundance. they're were made almost exclusive and publicf and also us a depressingly small number of technologies to be better off without. perhaps chemical weapons as a torture device. but we have not pulled out the blackball the idea that this civilization are the species that covers its with that probability.

low over half a century ago we discovered how to create nuclear weapons. it turns out to build an atomic bomb you need these difficult to obtain materials the only way to get done is to have the big facility their expensive in takes energy but instead there is no way to run the sat but u.s. simple message like baking bread in the microwave oven we know it prohibits that is physically prohibits that is physically impossible but before wed how that would turn out? so to make sure it was

cooperative but if we keep pulling out of balls to leave the black ball in there we will eventually pull it out. what we don't have is the inability to put it back it is very hard to achieve that. so probably that would have been the end of human civilization with our ancestors. modern civilization would have been impossible or go back to the stone age and even if after a few hundred years if we could claw our way back up again with technological advancement we could unleash the same treatment again. -- daemon again but is that possible or probable that

could be such a blackball? i could think of a few different areas where existential could observe i will talk about ai but there are others that can increase our abilities with the enormous upsidekv obviously. but with the nanotechnology is not that version but through the carbon powders but that futuristic version of that that is atomic the precise machinery. and that existentialist risk doesn't just cause extinction but from an optimal state.

but if we change these parameter values with the radical advances of surveillance or the discovery of a psychological technique to manipulate desires could make it easier or harder a small group of people to control a large group. we don't have the level of political science to predict if you handle this parameter. maybe it could nip insurgencies in the bud. human modification a lot of the unknown but if we take a setbackmk to ask the question it is striking that now

would have been listed if not a.i. they did not even have that concept. but some worry about tendencies but for the most part their own their recent discoveries so there may be additional existential that is on the radar to add the expected value to look that it is impossible to do something about.5z if we combine these with the notion to where development is a the micro level with

the structures the choice is that individuals make but it seems impossible fed general purpose technology and also intermediate leading up to that with practical uses in the performance of time and technology we will indent stuff -- invent it metaphorically if you have the big box that is empty then you pour in the sand that determines which area you find is where you get the capabilities but eventually it becomes obvious so issue previously

with the enormous upside so what attitude you want to take? what kind of response? so there are two basic approaches but the attitude that is best expressed i think it will go faster why should i care about the world if i am dead and1 gone? this increases the chance of more technological advanced future. so you have to be clear if the question is from the perspective what should i hope for?

technological advancements? said is right. -- that is right to have that life span to have good jupiter sized brain than clearly that will not have been in the course of defense for defaults is we will die readjust wrought ting away.hh but unless a radical shake-up pokers -- joker's something radical past two ocher so that is to hope it is quickly as possible. but to even aside from this extreme lifestyle we should probably help for faster

technology. but ask a different question what is best personally? it is closer to the idea expressed of the technological development of powerful technologies and accelerated development especially those their reduced the existential risk from other technologies. other than a blank check to make everything happen sooner we should think of the different dimensions. it is not the hypothetical technology do we want to develop that or not?

the answer might be pre-will develop but it is on the margin does that come sooner or later? to think we have some ability to affect that timing otherwise it is just wasted with@klz impact at all. so by promoting the field to move things around a couple of months later and stayed like a small change to make it is a existential rest. and in many cases it might

make it a huge difference so take the three earlier flight a.i. each has submitted existentialist risks on one trajectory and then we get lucky then redevelop nanotechnology. then we get through that and finally with the machine super intelligence but it is but if it got them in a different order we can have the existential risk but we

still have to hope through the radar but the fit is defeated it to help eliminate or reduce the other. the whole definition with all things considered with a macro strategic issues but the timing may give us more insight rather we should have technology x or not. so we can propose a replacement of the sustainability so traditionally people think in terms of achieving space

we could remain in that space with the resources that are replenished only at the rate their broken-down. but to suggest instead'v to achieve this space we should pursue a trajectory with which we could continue for a long time in a desirable direction if that is best done by increasing or decreasing to convert a lot of fuel to maximize sustainability before it runs out of fuel and crashes

but then to maximize the thrust then you have more sustainable path you can continue. but we should maximize the consumption of fossil fuels but with technology capability and some kind of measure that the world can solve global coordination problems such as avoiding war and arms races destroying public goods than the third the insight or the ability to know how to use other resources better truly worthwhile. to really have] the point where we have huge amounts of each quantity u.s. super

due for advanced technology but make sure you have a deep insight. to have more mindless entertainment so all the end point may reach up there but that is the rocket ship but it might weaken have these advanced technological capabilities it is only after reversed daughter act together to achieve global coordination. so so much for the framework for the micro strategic issues although briefly into the specific issue it will

likely be to the era of super intelligence this will be the most important thing to happen but looking at other existential catastrophes' to even have the shot

so to some extent it is not the question of intelligence furs but to think how to enhance that will initially become technologically feasible is through genetic engineering there are other ways but it is unlikely that if you injected it would make you smarter. i think it would have devolved. but

sufficient degree but sequencing has been high enough to make it possible to conduct large-scale studies with large populations but it is not falling where studies are on the ticket but that is because the variations is not due to a few different genes it is hundreds or thousands of genes of to discover of large effect you need a large sample than no other technology would be required in the context of

fertilization with normal fertility treatmentij to look at a visible abnormality you can also screen for certain disorders for down's syndrome but not positive complex behavior to become a feasible. this technology is combined which is the ability to drive sperm from the stem cells. it could produce the initial set of embryos to select the best with the highest trade value then use that from which you could drive to

again meet to drive another generation of embryos. bid is the collapse from 20 or 30 years and in the sense to persuade people to change their breeding patterns now science does this in a petri dish without changing crew mates with home but this technology has been used with mice with gamete but is not for use in human population. this analysis along with my colleague with the effects of different levels of selection. if you select the most promising for intelligence

their new gatt may be for i.q. points may be a love letter 12 or one in 1,000 you can see is deeply diminishing returns increasing the population. but instead of doing the one shot selection you do repeated selections then you don't get the diminishing returns may be as many as 65 at that level to bring out from all of human history that the scientist would be

and how far you would go it is an chartered territory but through these types of biological enhancements there is no reason to think this species is the smartest one that could exist. it is probably the dumbest capable to treat technological civilization as our ancestors got slowly smarter that they could do this then they did it. but no reason to think there is not a huge space above us. so that creates a weak form of super intelligence but ultimately the potential for processing is the point it looks dangerous for those

that are smarter than us for progress to keep up with the machines to keep one step ahead for good if we do pursue the biological to be more capable with smarter scientist from artificial intelligence and computer science is still seems like it may be beneficial but not because of the delay the takeover by machines but something more competent when we create these super intelligent machines. unsaid of these processor i

will not talk about that now but the computer interface i don't think that his reaction is. it will be difficult to enhance the capabilities to implanted when your body. if you can access just to think about it but i can access google already died and made surgery to do so. most of the benefits come through this are achieved by this same device by accusing your eyeballs. [laughter] one hundred gigabytes per second into your brain your visual cortex submit is hard

to beat that but the bottleneck is not squeezing sensory data but to try to extract those features. but i the time to have intelligence my guess is the electronic parts is so smart that that leaves for the machine as the main path so for the public at large that they are milestones but of course, underneath there is more progress their dead

different techniques that have been developed. to produce machine intelligence through this approach to clear the need something more to have fundamental new discoveries architectural that are missing. some of this magnitude but for those that have been developed in the last six years since we had computers. that is the lifetime of some people that are around today. it is not really that long

but if you provide another century to measure the performance of machine intelligence against human intelligence that made as significant contribution over the past couple of decades due to hardware getting faster. that seems to hold. but there is another path to machine intelligence which is to have the existence proved like the human brain and if it is too difficult to figure out from scratch with mathematics we could

try to reverse engineer the human brain that is at different levels of ambition but on the one hand it could be disorganized with the algorithms with completely synthetic artificial algorithms to copy a particular human brain so the idea is to take a particular brain and freeze it and slice it and put those through a microscope to take pictures used the image recognition to extract three dimensional connectivity matrix and combine that with the models

with the neuron functions run that through the powerful computer. this is something we are far away from doing now this is really advanced technology. but it doesn't require any new concept preachers to understand how thinking works to produce machine intelligence to understand the parts of the brain and using technology to build. you can see epigraphic the cross-section at the bottom there is a block on top of one another, that is the art of image recognition.

we already have microscopes but for that entire brain would take roughly forever. [laughter] in the picture looks very nice but they add up with that narrow competition models that is a lot of incremental work required to make this feasible celaeno it is not just around the corner. is passed to be decades out. made the blunder but not next year. there is some small chance you can make the expected medical breakthrough.

but the fact there is the multiple task if it turns out it is just too harsh for us to crack we can wait to have us smarter version but to increase the probability it will be reached. but then researchers complained so with those techniques that developed they use of all the software nidal just like a failure to say that has not yet gone all the way.

so when will this point have been? we did a survey withk0 experts by what year the think we will have a human level machine intelligence? the answer was depending on which route we asked. the footnote to condition allies without a collapse of global civilization so maybe you have to push that up. but my own view is sensible there is a large uncertainty could have been much sooner or take much longer. if there is a 90% probability they said to thousand 75 that might be overconfident that to put

more than 10 percent probability. if and when we do reach a human level machine intelligence then we get a superd/ intelligence? you can see the results there i am very agnostic on human level machine intelligence. estimate reaches that point that they get their super -- very soon after bite if they distinguish into one overall level but the question how far we are now from a human equivalent how quickly will

machines leave us in the dust? and something hinges on this question. if we have a fast take up up, for a start it seems like it will not be possible to have new solutions if we have the favorable outcome is cause of the preparations if on the other hand, for decades or centuries incrementally expand that capable levels of human

dependent on how fast the takeoff will be. with a fast take up you can have the world order the highest level of decision making agency where going from human level to super intelligence in hours or weeks than it would be one product before the next even started. but it is rare it is so close to one another maybe the leader has a few months. the faster the takeoff that you have a system in the world we're it is even close and it is likely the first

super intelligence can be powerful. it is not that the muscles are strong for but that enables us with these technologiest6 that makes us the dominant species to the point now even though guerrillas are much stronger it depends on what we do. with this scenario of a of an intelligence to quickly achieve maturity a telescope of the future the medical achievements we could not achieve over 35 years have been very quickly so you have a potentially extremely powerful agent to shape the future that could hinge on

what goes on and we can talk more about that but it is more likely to have a multi poler outcome. with the systems with a comparable level of culpability but at one point it could just lay down the law. with that outcome of very different set of concerns. not less serious though. you then have economic competitive forces coming down with a population of pivotal mines and depending on theñv assumptions with the population explosion people of make copies of them that

equals the subsistence like the price of car rental but that his firm biological because we need houses to live-in it can create models where it becomes impossible where we rely on income from capital. the question is if we could preserve these indefinitely that gradually become much faster than any rate the evolutionary outcome0 might

not contain anything that looks like a human with that song and dance it could turn now they're just optimized to bring in output from the economy. so it needs to be thought through either way but it does look like the fulcrum on which the future hinges with the arrival of super intelligence is a good candidate to what the fulcrum might look like. so now let's open it up to have a discussion. thanks. [applause] i will briefly mention the microphone is down here you can line up and get ready with your questions.

>> go-ahead and line up. >> it is extremely hard for me to take your presentation is seriously. this idea of the singularity of machines becoming intelligent comes from the 1950's and througho those three parts depots the first is genetic. you propose for example, excel rating of the selection. if you do with that way presuming that we do arrive at said genes that are conducive to intelligence which we have not done that yet. so even if you did you leave

out genetic defects. second neuroscience project is exactly the one that henry is getting 1 billion euros for at the moment they signed a petition saying is very rich and a waste of money. you can check this on the web. it is hopelessly naive fan the third is interesting that a.i. is most likely to succeed it is only for a short period of time but i do not see a single technique developed after 1980. we do have different names like the singular narrowed net but are you selling books or is this a serious academic thesis? >> will take those questions

the biological and genetic selection it is complications related to genetics it might be you can only capture a part but even by that you could get some of the way to enhance the genetic predispositions. if you also develop to understand with sedation you could get further but without making that assumption it is part of the way. i said nothing with henry and i have no opinion is that is worthwhile or not. but i point out that at some point we are advancing technologies in these fields it is consistent with a

complete waste of money to do it anytime soon. i am not sure even if there was a prospect of succeeding so i'm kind of happy it does not look promising. can with the recent discoveries with my insight i think deep learning is fairly recent. and conditional like big impression is that progress has probably continued more or less at a constant rate it is difficult to measure but there does not seem to

be any slacking off there is a lot of enthusiasm and interest to acquiring a.i. companies like deep mind in london was just acquired by google for $400 million no competition with facebook trying to scoop up the competition. . .