# Why We Need Friendly AI

> An annotated edition of this paper on Fermat's Library, where scientists and readers explain its ideas, equations and history in the margins. Annotated edition: https://fermatslibrary.com/s/why-we-need-friendly-ai

- **Authors:** L. Muehlhauser & N. Bostrom
- **Published:** 2014
- **Topic:** Computer Science
- **Original:** https://s3-us-west-1.amazonaws.com/fermatslibrary/uploads%2F1485896422447-q19hw9acrfr-dde2e20cb433117ae66fc13b4e2ebaf6%2Fbostrom_whyfriendlyai.pdf
- **Pages:** 7
- **Annotations:** 11 comments in 9 threads
- **Featured:** Fermat's Library Journal Club
- **Annotated edition:** https://fermatslibrary.com/s/why-we-need-friendly-ai

## Annotations

### Annotation 1 (page 1)

Beside the passage: “The human level of intelligence is an evolutionary accident – a small basecamp on a vast mountain side, far below the highest ridges of intelligence allowed by physics. If we were visited by extraterrestrials, these beings would almost certainly be very much more intelligent and technologically advanced than we are, …”

**Annotation** by Micael Oliveira (2017-02-27):

> >**TL;DR:**
> 1. The transition to an AI controlled era could be more abrupt than we expect
> 2. Human-level AIs could have goals of self-improvement independent of human values
> 3. Confinement of potentially dangerous AIs does not solve the problem
> 4. We need to code Morals and Ethics into AIs but leave degrees of freedom for morals to evolve
> 5. Prepare the transition as soon as possible by raising awareness and focusing research on friendly AIs.

### Annotation 2 (page 1)

Beside the passage: “But aliens are unlikely to make contact anytime soon. In the near term, it seems more likely we will create our intellectual successors. Computers far outperform humans in many narrow niches (e.g. arithmetic and chess), and there is reason to believe that similar large improvements over human performance are possible …”

**Annotation** by Micael Oliveira (2017-02-27):

> **Go, considered as the most complex board game ever invented and famously difficult for computers to crack has already been mastered by computers.**
>
> In 2015 AlphaGo first defeated a professional human player, and then went on to beat one of the Go world's top players, Lee Sedol.
>
> You can read more about it here:
> - [Google reveals secret test of AI bot to beat top Go players](http://www.nature.com/news/google-reveals-secret-test-of-ai-bot-to-beat-top-go-players-1.21253)
>
> - [Google’s AI Wins Fifth And Final Game Against Go Genius Lee Sedol](https://www.wired.com/2016/03/googles-ai-wins-fifth-final-game-go-genius-lee-sedol/)
>
> - [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo)

### Annotation 3 (page 1)

Beside the passage: “Though some doubt that machines can possess certain mental properties like consciousness, the absence of such mental properties would not prevent machines from becoming vastly more able than humans to efficiently steer the future in pursuit of their goals. As Alan Turing wrote, ‘...it seems probable that once the …”

**Annotation** by Micael Oliveira (2017-02-27):

> *“Let us now assume, for the sake of argument, that these machines are a genuine possibility, and look at the consequences of constructing them. To do so would of course meet with great opposition, unless we have advanced greatly in religious toleration from the days of Galileo. There would be great opposition from the intellectuals who were afraid of being put out of a job. It is probable though that the intellectuals would be mistaken about this. There would be plenty to do, trying to understand what the machines were trying to say, i.e. in trying to keep one’s intelligence up to the standard set by the machines, for it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler’s ‘Erewhon’.”*
>
> **– Alan Turing, “Intelligent Machinery, A heretical theory”**
>
> You can read Turing's paper here: [Intelligent Machinery, A heretical theory](http://fermatslibrary.com/s/intelligent-machinery-a-heretical-theory)

### Annotation 4 (page 1)

Beside the passage: “There is, of course, a risk in passing control of the future to machines, for they may not share our values. This risk is increased by two factors that may cause the transition from”

**Annotation** by Micael Oliveira (2017-02-28):

> Renowned scientists, such as Stephen Hawking and Stuart Russell believe that AI poses a significant threat to the human species. If human-level AIs have the ability to re-design and improve themselves at an ever-increasing rate then we could witness an **unstoppable "intelligence explosion" could lead to human extinction.**

### Annotation 5 (page 2)

Beside the passage: “human control to machine control to be quite sudden and rapid: the possibilities of computing overhang and recursive self-improvement.”

**Annotation** by Micael Oliveira (2017-02-27):

> Human-level AIs might not share human values. The risks associated with passing control to human-level AIs are aggravated by the two following factors:
>
> - **Computing Overhang:** Assume that computing power continues to increase according to Moore's Law and it takes us more time than expected to create human-level AIs. When we finally develop the human-level AIs there could be an excess computing power available that would allow AIs quickly replicate themselves and thus surpass the human population very abruptly.
>
> - **Recursive Self-improvement:** Assume that AIs have goals to preserve themselves and self-improve. When AIs start replicating and improving themselves their intelligence level could quickly surpass human-level intellingence. Their level of intelligence could be so far superior to ours that it would be impossible for us to negotiate with them as it is impossible for chimpanzees to negotiate with humans. A superintelligent AIs will want to pursue their own goals independently of humans.

**Reply** by Rens van der Heijden (2017-03-01):

> I think that the formulation of this risk is often made a bit too strong. Yes, the risk exists, but the development and deployment of a general AI would not automatically imply all of this. That assumes that several well-known challenges and limitations in distributed systems are solved by general AI. Assuming that such general AI is deployed, there is no guarantee that a population of (small) independent AIs is as "good" as a single super-intelligent and highly distributed AI (running everywhere at the same time, ala Skynet). A highly distributed AI would need to deal with limitations such as the [CAP Theorem](https://en.wikipedia.org/wiki/CAP_theorem), while a population of AIs would need to deal with potentially malicious instances of the "AI population" (formally speaking, they'd need things like [Byzantine Fault Tolerance](https://en.wikipedia.org/wiki/Byzantine_fault_tolerance)).
>
> I'm curious to know whether there is any conceptual work in this direction (what are the fundamental computational limitations of general AI?). Is there actual research happening in this area? I'd be very interested to read more about that.

### Annotation 6 (page 3)

Beside the passage: “But because we will create our own successors, we may have the ability to influence their goals and make them friendly to our concerns. The problem of encoding human (or at least humane) values into an AI’s utility function is a challenging one, but it may be possible. If we can build such a ‘Friendly AI,’ we may not …”

**Annotation** by Micael Oliveira (2017-02-27):

> Elon Musk and Sam Altman and a group of Computer Scientists decided to found OpenAI with the goal of creating **friendly AIs**. OpeAI is a non-profit artificial intelligence research company whose mission is to build safe and friendly AIs. Their focus is creating a positive long-term human impact and ensure AI's benefits are as widely and evenly distributed as possible.
>
> You can learn more about here: [About OpenAI](https://openai.com/about/)

**Reply** by Rens van der Heijden (2017-03-01):

> I'd like to leave a comment regarding the use of a *utility function* as the exclusive guideline to what a machine will do. To me this seems to be compatible only to a strictly utilitarian world view, and I feel as though the all-consuming general AI is just Nozick's [utility monster](https://en.wikipedia.org/wiki/Utility_monster) thought experiment made real. Which is to say that a utility function is probably not the ideal way to build *general AI* (although it works very well for specific AI applications, such as classification and learning algorithms).

### Annotation 7 (page 3)

Beside the passage: “First: why not just keep potentially dangerous AIs safely confined, e.g. without access to the internet? This may sound promising, but there are many complications. 3 In general, such solutions would pit human intelligence against superhuman intelligence, and we shouldn’t be confident the former would prevail. …”

**Annotation** by Micael Oliveira (2017-02-27):

> **One proposed solution for a better transition to human-level AIs is Confinment, i.e. not giving AIs access to the internet.** Some arguments against the confinement of AIs are:
> 1. It depends on all AI-research groups being careful on keeping their AIs confined
> 2. When governments understand the power of AIs and the advantage it represents they will incentivize developemtn speed over safety.

### Annotation 8 (page 4)

Beside the passage: “Second: some have suggested that advanced AIs’ greater intelligence will cause them to be more moral than we are; in that case, who are we to protest when they do not respect our primitive values? That would be downright immoral!”

**Annotation** by Micael Oliveira (2017-02-27):

> AIs could have none or a completely different definition of morality. We need to code some of our values in the AIs but we do not want to dictate all of their values because morals tend to evolve and improve with time. A possible solution to this problem would be to give machines an algorithm for figuring out what human values would be if we were wiser. This is an interesting philosophical problem and it could be a good solution for machine moral/ethics.

### Annotation 9 (page 5)

Beside the passage: “Third: others object that we are too far from the transition from human to machine control to work on the problem now. But we must remember that economic incentives favor development speed over development safety. Moreover, our scientific curiosity can sometimes overwhelm other considerations such as safety. To quote …”

**Annotation** by Micael Oliveira (2017-02-27):

> We should raise awareness about AI and the threats it poses to humanity, focus on developing friendly AIs and prepare the transition as soon as possible. This type of work as already started with research institutions such as the [Machine Intelligence Research Institute in Berkeley](https://intelligence.org/) and [Future of Humanity Institute at Oxford University](https://www.fhi.ox.ac.uk/) and other companies like [Open AI](http://openai.com).

## Full text

Extracted from the PDF. [Annotation N] marks where each annotation above sits.

#### Page 1

http://journals.cambridge.orgDownloaded: 21 Apr 2014 IP address: 163.1.72.155

WHY WE NEED FRIENDLY AI Luke Muehlhauser and N ick Bostrom

Humans will not always be the most intelligent agents on Earth, the ones steering the future. What will happen to us when we no longer play that role, and how can we prepare for this transition?

The human level of intelligence is an evolutionary acci- [Annotation 1] dent – a small basecamp on a vast mountain side, far below the highest ridges of intelligence allowed by physics. If we were visited by extraterrestrials, these beings would almost certainly be very much more intelligent and technologically advanced than we are, and thus our future would depend entirely on the content of their goals and desires.

But aliens are unlikely to make contact anytime soon. In the near term, it seems more likely we will create our intellectual successors. Computers far outperform humans in many narrow niches (e.g. arithmetic and chess), and there is reason to believe that similar large improvements over [Annotation 2] human performance are possible for general reasoning and technological development.

Though some doubt that machines can possess certain mental properties like consciousness, the absence of such mental properties would not prevent machines from becoming vastly more able than humans to efficiently steer the future in pursuit of their goals. As Alan Turing wrote, ‘...it seems probable that once the machine thinking method [Annotation 3] has started, it would not take long to outstrip our feeble powers... At some stage therefore we should have to expect the machines to take control...’

There is, of course, a risk in passing control of the future to machines, for they may not share our values. This risk is [Annotation 4] increased by two factors that may cause the transition from

doi:10.1017/S1477175613000316 # The Royal Institute of Philosophy, 2014 Think 36, Vol. 13 (Spring 2014)

Think Spring 2014

† 41

#### Page 2

http://journals.cambridge.orgDownloaded: 21 Apr 2014 IP address: 163.1.72.155

human control to machine control to be quite sudden and rapid: the possibilities of computing overhang and recursive [Annotation 5] self-improvement.

What is computing overhang? Suppose that computing power continues to double according to Moore’s law, but figuring out the algorithms for human-like general intelligence proves to be fiendishly difficult. When the software for general intelligence is finally realized, there could exist a ‘computing overhang’: tremendous amounts of cheap computing power available to run human-level artificial intelligences (AIs). AIs could be copied across the hardware base, causing the AI popul ation to quickly surpass the human population. These digital minds might run thousands or millions of times faster than human minds. AIs might have further advantages, such as superior communication speed, transparency and self-editability, goal coordination, and improved rationality.

And what is recursive self-improvement? We can predict that advanced AIs will have instrumental goals to preserve themselves, acquire resources, and self-improve, because those goals are useful intermediaries to the achievement of almost any set of final goals. Thus, wh en we build an AI that is as skilled as we are at the task of designing AI systems, we may thereby initiate a rapid, AI-motivated cascade of selfimprovement cycles. Now when the AI improves itself, it improves the intelligence that does the improving, quickly leaving the human level of intelligence far behind.

A superintelligent AI might thus quickly become superior to humanity in harvesting resources, manufacturing, scientific discovery, social aptitude, and strategic action, among other abilities. We might not be in a position to negotiate with it or its descendants, just as chimpanzees are not in a position to negotiate with humans.

At the same time, the convergent instrumental goal of acquiring resources poses a threat to humanity, for it means that a superintelligent machine with almost any final goal (say, of solving the Riemann hypothesis) would want to take the resources we depend on for its own use. Such

Muehlhauser and Bostrom Why We Need Friendly AI

†

42

#### Page 3

http://journals.cambridge.orgDownloaded: 21 Apr 2014 IP address: 163.1.72.155

an AI ‘does not love you, nor does it hate you, but you are made of atoms it can use for something else’. 1 Moreover, the AI would correctly recognize that humans do not want their resources used for the AI’s purposes, and that humans therefore pose a threat to the fulfillment of its goals – a threat to be mitigated however possible.

But because we will create our own successors, we may have the ability to influence their goals and make them friendly to our concerns. The problem of encoding human [Annotation 6] (or at least humane) values into an AI’s utility function is a challenging one, but it may be possible. If we can build such a ‘Friendly AI,’ we may not only avert catastrophe, but also use the powers of machine superintelligence to do enormous good.

Many scientific naturalists accept that machines can be far more intelligent and powerful than humans, and that this could pose a danger for the things we value. Still, they may have objections to the line of thought we have developed so far. Philosophe r David Chalmers has responded to many of these objections; 2 we will respond to only a few of them here.

First: why not just keep potentially dangerous AIs safely confined, e.g. without access to the internet? This may [Annotation 7] sound promising, but there are many complications. 3 In general, such solutions would pit human intelligence against superhuman intelligence, and we shouldn’t be confident the former would prevail. Moreover, such methods may only delay AI risk without preventing it. If one AI development team has built a human-level or superintelligent AI and successfully confined it, then other AI development teams are probably not far behind them, and these other teams may not be as cautious. Governments will recognize that human-level AI is a powerful tool, and the race to be the first nation with such a great advantage may incentivize development speed over development safety. (Confinement measures may, however, be useful as an extra precaution during the development phase of safe AI.)

Think Spring 2014

† 43

#### Page 4

http://journals.cambridge.orgDownloaded: 21 Apr 2014 IP address: 163.1.72.155

Second: some have suggested that advanced AIs’ greater intelligence will cause them to be more moral than we are; in [Annotation 8] that case, who are we to protest when they do not respect our primitive values? That would be downright immoral!

Intelligent search for instrumentally optimal plans, however, can be performed in the service of any goal. Intelligence and motivation are in this sense logically orthogonal axes along which possible artificial intellects can vary freely. The imputed connection between intelligence and morality is therefore sheer anthropomorphism. (It is an anthropomorphism that does not even hold true for humans: it is easy to find humans who are quite intelligent but immoral, or who are unintelligent but thoroughly decent.)

Economist Robin Hanson suggests that inter-generational conflicts analogous to the ones that could arise between humans and machines are common. Generations old and new compete for resour ces, and the older generation often wants to control the values of the younger generation. The values of the younger generation end up dominating as the older generation passes away. Must we be so selfish as to insist that the values of Homo sapiens dominate the solar system forever?

Along a similar line, the roboticist Hans Moravec once suggested that while we should expect that fu ture robotic corporations will eventually overrun humanity and expropriate our resources, we should th ink of these robotic descendants as our ‘mind children.’ Framed in this way, Moravec thought, the prospect might seem more attractive.

It must be said that a scenario in which the children kill and cannibalize their parents is not everyone’s idea of a happy family dynamic. But even if we were willing to sacrifice ourselves (and our fellow human beings?) for the sake of some ‘greater good,’ we would still have to put in hard work to ensure that the result would be something more worthwhile than masses of computer hardware used only to evaluate the Riemann hypothesis (or to calculate the decimals of pi, or to manufacture as many paperclips as possible, or some ot her arbitrary goal that might be easier to specify than what humans value).

Muehlhauser and Bostrom Why We Need Friendly AI

†

44

#### Page 5

http://journals.cambridge.orgDownloaded: 21 Apr 2014 IP address: 163.1.72.155

There is, however, one good reason not to insist that superhuman machines be made to share all our current values. Suppose that the ancient Greeks had been the ones to face the transition from human to machine control, and they coded their own values as the machines’ final goal. From our perspective, this would have resulted in tragedy, for we tend to believe we have seen moral progress since the Ancient Greeks (e.g. the prohibition of slavery). But presumably we are still far from perfection. We therefore need to allow for continued moral progress.

One proposed solution is to give machines an algorithm for figuring out what our values would be if we knew more, were wiser, were more the people we wished to be, and so on. Philosophers have wrestled with this approach to the theory of values for decades, and it may be a productive solution for machine ethics.

Third: others object that we are too far from the transition from human to machine control to work on the problem now. But we must remember that economic incentives favor development speed over development safety. [Annotation 9] Moreover, our scientific curiosity can sometimes overwhelm other considerations such as safety. To quote J. Robert Oppenheimer, the physicist who headed the Manhattan Project: ‘When you see something that is technically sweet, you go ahead and do it and you argue about what to do about it only after you have had your technical success. That is the way it was with the atomic bomb.’ 4 Still, one might ask: What can we do about the problem of AI risk when we know so little about the design of future AIs? For a start, we can do the kind of work currently performed by the two research institutes currently working most directly on this difficult problem: the Machine Intelligence Research Institute in Berkeley and the Future of Humanity Institute at Oxford University. This includes:

1. Strategic research. Which types of technological

development are risk-increasing or riskdecreasing, and how can we encourage

Think Spring 2014

† 45

#### Page 6

http://journals.cambridge.orgDownloaded: 21 Apr 2014 IP address: 163.1.72.155

governments and corporations to shift funding from the former to the latter? What is the expected value of certain kinds of research, or of certain kinds of engagement with governments and the public? What can we do to reduce the risk of an AI arms race? How does AI risk compare to risks from nuclear weapons, biotechnology, near earth objects, etc.? Can economic models predict anything about the impact of AI technologies? Can we develop technological forecasting methods capable of giving advance warning of the invention of AI? 2. Technical research. Can we develop safe

confinement methods for powerful AIs? How can an agent with a desirable-to-humans utility function maintain its desirable goals during updates to the ontology over which it has preferenc es? How can we extract a coherent utility function from inconsistent human behavior, and use it to inform an AI’s own utility function? Can we develop an advanced AI that will answer questions but not manifest the dangerous capacities of a superintelligent agent? 3. Raising awareness . Outreach to researchers,

philanthropists, and the public can attract more monetary and human capital with which to attack the problem.

The quest for AI has come a long way. Computer scientists and other researchers must begin to take the implications of AI more seriously.

Luke Muehlhauser is Executive Director of the Machine Intelligence Research Institute. Nick Bostrom is Director of the Future of Humanity Institute at the University of Oxford. luke@intelligence.org

Muehlhauser and Bostrom Why We Need Friendly AI

†

46

#### Page 7

http://journals.cambridge.orgDownloaded: 21 Apr 2014 IP address: 163.1.72.155

Notes

1 E. Yudkowsky, ‘AI as a positive and a negative factor in global risk’, Global Catastrophic Risks (eds) N. Bostrom and M. Cirkovic (New York: Oxford University Press, 2008).

2 D. Chalmers, ‘The Singularity: a reply to commentators’, Journal of Consciousness Studies, vol. 19, nos. 7–8 (2012), 141–167.

3 S. Armstrong, A. Sandberg, N. Bostrom, ‘Thinking inside the box: Using and controlling Oracle AI’, Minds and Machines, vol. 22, no. 4 (2012), 299 –324.

4 Robert Jungk, Brighter than a Thousand Suns: A Personal History of the Atomic Scientists, trans. Lames Cleugh (New York: Harcourt Harvest, 1958), 296.

Think Spring 2014

† 47

---

Source: [Fermat's Library](https://fermatslibrary.com/) — https://fermatslibrary.com/s/why-we-need-friendly-ai. More annotated papers: https://fermatslibrary.com/llms.txt
