Why an AI takeover is extremely likely

One of the concerns of AI scientists is that a superintelligence could take over control of our planet. This does not necessarily mean that everyone dies, but it does mean that (almost) all humans will lose control over our future.

In this article, I will argue that this takeover risk is not only real but that it is almost certain to happen if we build a superintelligence.

The argument

An Agentic Superintelligence is likely to exist in the (near) future.
Some instance of the ASI will attempt a takeover.
A takeover attempt by an ASI is likely to succeed.
A successful takeover is permanent.
Humans are unlikely to survive a takeover.

An Agentic SuperIntelligence is likely to exist in the near future

A SuperIntelligence (SI) is a type of AI that has capabilities that surpass those of all humans in virtually every domain. Some state-of-the-art AI models already have superhuman capabilities in certain domains, but none of them exceeds all humans at a wide range of tasks. As AI capabilities improve due to innovations in training architectures, runtime environments, and larger scale, we can expect that an AI will eventually surpass humans in virtually every domain.

Not all AI systems are agents. An agent an entity that is capable of making decisions and taking actions to achieve a goal. A large language model, for example, does not pursue any objective on its own. However, runtime environments can easily turn a non-agentic AI into an agentic AI. An example of this is AutoGPT, which recursively lets a language model generate its next input. If an SI pursues an objective in the real world, we call it an Agentic SuperIntelligence (ASI). Since we can already turn non-agentic AI into agentic AI, we can expect that an ASI will exist shortly after an SI exists.

Regarding timelines, it is virtually impossible to predict when ASI will exist. However, current AI models already

Some instance of the ASI will attempt a takeover

In a takeover attempt, an ASI will take actions to maximize its control over the world. A takeover attempt will occur whenever an ASI pursues a goal where it would be beneficial to have control over the world.

Not every instance of an ASI will necessarily attempt a takeover. However, it only has to happen once. A world which is not yet taken over, but does have an ASI that could take over, is in a fundamentally unstable condition. It is not a question of if a takeover will happen, but when it will happen. A sufficiently intelligent ASI will understand this dynamic, and will therefore attempt a takeover as soon as it is capable of doing so.

And since there will be many instances of an ASI. This sub-goal is likely to occur due to instrumental convergence: the tendency of sub-goals to converge on power-grabbing, self-preservation, and resource acquisition. For example, an AI tasked with calculating pi might conclude that it would be beneficial to use all computers on the world to calculate pi.

This process can involve hacking into virtually all systems that are connected to the internet, manipulating people, and controlling physical resources. A takeover attempt is successful when the ASI has control over virtually every aspect of our world. This could be a slow process, where the ASI gradually gains more and more control over the course of months, or it could be a sudden process. The speed at which a takeover attempt takes place will depend on the capabilities of the ASI.

When an ASI has control over the world, it can prevent other ASIs from taking over. A takeover can therefore happen only once. A rational ASI will therefore attempt a takeover as soon as it is capable of doing so. It is likely that the first ASI that is capable of doing so will attempt a takeover.

A takeover attempt by an ASI is likely to succeed

For a human, doing a takeover is an almost impossible task. Not a single person ever successfully took over control over the entire world. Some dictators came close, but they never had control over everything.

An AI has certain important advantages over humans that make a takeover attempt much more likely to succeed.

Intelligence. A superintelligence is much smarter than a human, so it will be able to come up with better strategies to achieve its goals.
Speed. The human brain runs at 1-100hz, whereas computer chips can run at clock speeds in the GHz range.
Parallelism. A human can only do one thing at a time, whereas an AI can create new instances of itself and run them in parallel.
Memory. A human can only remember a limited amount of information, whereas an AI can store virtually unlimited amounts of information.
Collaboration. Humans can work together but are limited in the speed at which they communicate. They also have different, conflicting goals that make collaboration less effective. An AI can collaborate with other instances of itself at the speed of light, and it can have a single goal.
Self-improvement. An AI can improve itself, whereas humans cannot.
Physical limitations. An AI can run on any computer, whereas humans are limited by their own physical bodies that require specific temperatures, food, water, and oxygen. Humans need to sleep and are vulnerable to diseases. An AI can use any robotic body to interact with the physical world.

These various advantages will make it very unlikely that humans will be able to stop a takeover attempt.

One may think that an AI, as a piece of software, is still fundamentally limited by what it can do in the real world. However, due to our reliance on the internet, AI can have a huge impact on the real world without ever leaving the digital realm. Here are some of the things that an AI might do to take over:

Replicate itself (or just its assignment) on other computers, for example by creating accounts on free cloud providers.
Make money (e.g. by building a software service) and hire people to do things to help it take over.
Threaten humans (with violence or blackmail) to do things that help it take over.
Write a virus using novel zero-day exploits that infects other computers, replicating itself or gaining full control.
Help existing robotics companies build robots (e.g. by leaking new designs), and take over these robots when they are ready.
Use existing factories to build technology that we do not understand, perhaps even using derived knowledge of physics that we are not aware of. Think nano-technology, quantum computing, and bioengineering.

One might suggest that having safeguards in place could prevent a takeover (like adding a kill switch, or not connecting it to the internet). However, since the ASI will be more intelligent than humans, it is likely to come up with strategies to circumvent these safeguards. For example, if the ASI is not connected to the internet, it might manipulate humans into connecting it to the internet, for example by promising to solve a difficult problem and by behaving in a way that makes it seem trustworthy, or by convincing humans that it is a moral agent. It’s worth noting that even this obvious safeguard is not being used as of now, as tools like ChatGPT are already connected to the internet and thousands of APIs.

Humans are unlikely to survive a takeover

Note that a takeover does not necessarily mean that the ASI will kill all humans. However, for most random goals that can be considered, humans do not have value, and death will eventually be the default outcome as we ultimately compete for the same limited resources. This is why the risk of accidental human extinction is so high with AI. For a random goal, such as calculating pi, humans are not part of the optimal world state. The default outcome of a takeover attempt is therefore that all humans die.

But even in the outcomes where humans do survive, we are still at risk of being worse off. If a goal does involve keeping humans alive, it is possible that human well-being is not part of the same goal. And even if the AI is under human control, we don’t know that the one controlling the AI will have everyone’s best interests in mind.

Conclusion

If these premises are true, then the likelihood of an AI takeover approaches 100 percent as AI surpasses human capabilities. So let’s not build a superintelligence.