Superintelligence: Paths, Dangers, Strategies — Nick Bostrom — review

First glance at the book

As the name suggests, the book contains the state, dangers, and strategies of artificial intelligence today. It may sound intimidating, but in fact, it is a book that can be read comfortably as it does not overwhelm the reader with knowledge but emphasizing the philosophical structure. However, I should warn, if you are very early in the artificial intelligence concept, this book may not be for you.

Another warning about the book is, if you do not want to be frightened about our future, you would not want to read that sort of book.

“…As the fate of the gorillas now depends more on us humans than on the gorillas themselves, so the fate of our species would depend on the actions of the machine superintelligence. ”

The purpose of the book

The main purpose of the book is to inform the reader about the dangers of artificial intelligence and the ways we can avoid them. The book achieves this purpose by creating scenarios in which certain points are accepted. Main topics can be separated as follows.

Past developments and today’s capabilities

How to develop a superintelligent system and what are the types

Intelligence explosion and its effects, benefits, and dangers

The book affects the way we think about superintelligent systems in a negative way. While it informs us about superintelligence and its benefits (relatively rarely), it also clearly warns us about how dangerous AI can actually be.

Main arguments

From the most general perspective, the main argument is how it will change humanity if machines become smarter than humans. While discussing these, the book explains the possible dangers, moral issues, safety precautions, and the ways to develop such a superintelligent system.

Besides superintelligent systems’ dangers, there are a lot of huge benefits; however, in this book, the main approach is from the negative perspective and the aim is to paint a World with superintelligent systems being detrimental.

Superintelligence

To understand the subject better, at least we have to say a few words on superintelligence. There are 2 possible ways mentioned to “create” them. Basically, we can say that one of the ways is based on developing inspired by the principles of human intelligence and the other is based on the idea that fully imitating a human brain such as “Whole Brain Emulation”.

“Quality superintelligence: A system that is at least as fast as human mind and vastly qualitatively smarter.”

Due to the fact that computers do not manage to process enough data to reach its goal if it is not specific like playing chess or solving a problem (with today’s technology), it is hard to develop a human-like intelligence system. A possible approach to solve that problem is called the “child machine” which is named by Alan Turing. The principle here is to develop a machine that only has base knowledge and let it improve itself with its own experiences. Another possible solution is WBE(Whole Brain Emulation).

“We cannot expect to find a good child machine at the first attempt. One must experiment with teaching one such machine and see how well it learns…If he can trace a cause for some weakness he can probably think of the kind of mutation which will improve it.”

Dangers

In order to have an opinion about the dangers, here is the section for ones curious not just about the book but also about the subject. The damages are basically divided into “benign” and “malignant” failures and many of these failures are baning in the sense that they will not lead to existential catastrophe. To understand why “malignants” are “malignants”, here are a few types and scenarios.

“Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization”

Preverse instantiation

The one-sentence summary for that kind of superintelligence is, “a superintelligence discovering some way of satisfying the criteria of its final goal that violates the intentions of the programmers who defined the goal”.

“Final goal: Make us happy — Perverse instantiation: Implant electrodes into the pleasure centers of our brains”

Making the definition of the goal as specific as possible and pointing the possible paths to solving it may prevent us from that problem. However, it may cause other kinds of problems as follows.

Infrastructure profusion

In short, the Infrastructure profusion problem is the conversion of the accessible universe into infrastructure in order to reach a goal by a superintelligent system. As an example, Nick Bostrom suggests giving a goal to a superintelligence “making paperclips”. The problem is, the machine may not halt and keep redirecting resources to infrastructure.

Mind crime

The mind crime concept is explained by suggesting an AI system that contains morally relevant computations and treats them in a bad way. Thus there can be a situation like an AI system simulates humans with the purpose of learning human psychology and then quickly destroys them.

Strong points

The author, Nick Bostrom, who is a philosopher and known for his work on existential risk and human enhancement ethics, also has deep knowledge and good metaphors to give about superintelligence risks. He points the dangers in futuristic but also in a realistic way of painting a near-future World.

Dissapointments about the book

Since there are just a few visual elements in the book, some of the subjects are being hard to visualize. There might be more charts or graphs to show the correlation of variables mentioned in a specific subtopic.

The other deficient point of the book is, it might be simpler to talk to a wider group of people. However, it could be as simple as that considering the subject is that deep.

Last words

To sum up, the development of a superintelligent system has both benefits and possible harms. In order to minimize the risks of this technology as much as possible, the security problem must be solved first.

The last words not about the subject but the book are, it will probably be a template for almost all future books written about this subject and everyone who wants to get informed about the dangers of AI systems must read it.

“We would want the solution to the safety problem before somebody figures out the AI problem.”