The men building AI want us to fear it. We should – but not for the reasons they give

Is AI going to run wild and kill us all – or are the concerns lately voiced by the bosses of the AI titans just some kind of convenient business ploy on their part? Those bosses include Dario Amodei of Anthropic, which owns Claude; his rival Sam Altman of OpenAI, which owns ChatGPT and its descendants; and their competitor, SpaceX founder Elon Musk, who owns Grok. The debate has raged around the world for the past few days.

Dividing the world into two kinds of people never works, so we’ll go with three: this issue has divided the world into those who agree that AI may indeed kill us all, those who think Musk et al are out to fool us all, and those in the “don’t know/can’t say” camp. Our analysis leads us to think there ought to be a fourth camp: those who agree that the AI bosses may be trying to fool us all, and also that AI may kill us all. The two are not mutually exclusive.

Debates over AI safety are old, but the latest drama began with the public resignation of Jacob Coxon, an AI researcher at Anthropic who had earlier worked at OpenAI. In a viral tweet, Coxon, who worked on training AI models, wrote that both companies are “racing straight to superintelligence and gambling with our lives…Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources”.

Amodei responded, saying, “I agree with Jacob much more than I disagree with him.” Four days later, on September 12, he published an essay arguing that the AI industry should slow down. “We must slow the pace at which we improve the capabilities of AI models,” he wrote. To everyone’s surprise, Altman replied, saying, “I agree with Dario that we need to pace the frontier.” Then Musk pitched in, saying, “Dario is right.” Demis Hassabis, a Nobel laureate in chemistry who heads Google DeepMind, which runs Gemini, also agreed, saying, “Dario’s essay points towards the right path forward.”

The calls from the industry’s most influential voices have been backed by leading academic researchers. In his doomsday warning, Coxon also cited a more than 10 percent chance that AI could wipe out all humans within the next decade. Professor Geoffrey Hinton, a Nobel laureate in physics who is often called the “Godfather of AI”, told the BBC that “A 10 percent chance seems not an unreasonable estimate to me, but nobody really knows how to give a sensible estimate.”

The broad consensus among academic researchers and industry leaders at the cutting edge of AI therefore seems to be that superintelligent AI systems may indeed be developed, and that they may then escape human control and turn against us.

This might seem like a straightforward win for the AI doomers, given that these are the people building the AIs and presumably know them better than anyone else, but the other side has its champions too.

AI hype and puffery

Michael Burry, a famous investor portrayed by Christian Bale in the Hollywood film The Big Short, has said in posts on X and Instagram that it is “self-serving for OpenAI, Anthropic and other execs of big hyperscalers to talk of slowing things down”. While the term “hyperscalers” is generally used for large cloud service providers, Burry has used it for the AI giants that rely on sheer scale in computation and data to achieve their results. He argued that Large Language Models (LLMs) are not really Artificial Intelligence and will not become Artificial General Intelligence (AGI), so there is nothing to slow down. 

We currently use the term artificial intelligence for sophisticated chatbots such as Claude, Gemini, the various GPTs and DeepSeek, all of which are LLMs. These systems don’t reason to arrive at conclusions; they work by probabilistic pattern matching, essentially generating the most probable words to form sentences, one word at a time. They are trained on humongous amounts of data, and the data centres that run them need enormous amounts of power and water. They all consist of artificial neural networks that carry out millions of mathematical operations using advanced computer chips. In their fundamental operations, they only calculate probabilities.

Critics like Burry question whether this is really intelligence at all.

Burry also argued that the incumbents face competition, and that forcing a slowdown on everyone while they are still ahead would benefit them. Both Anthropic and OpenAI have Initial Public Offerings (IPOs) coming up. While Anthropic’s IPO is now expected sometime in November, OpenAI’s is likely to be next year. 

IPOs, Burry said, need “hype and puffery…we’re so awesome it could become dangerous is hype and puffery.” In his view, the slowdown talk is cover for a real slowing of growth that the companies cannot control.

AI ethics researcher Timnit Gebru, formerly co-lead of Google’s Ethical AI team, has long argued that the idea of superintelligence wiping out humanity is hype and puffery. Responding to the current debate, she has said that the talk of a “machine-god” that will wipe us out is meant to distract us from current, ongoing harms.

The use of AI in warfare to kill human beings is one example of such harm. Autonomous drones with AI targeting are already killing people in the Russia-Ukraine war. AI-assisted targeting was also used by the US in its attack on Iran, and by Israel in Gaza. On February 28, 2026, the first day of the joint US-Israeli war against Iran – the American part of which was named Operation Epic Fury – a strike on a primary school in Minab killed at least 156 people, including 120 children, according to Iranian officials. A UN-backed fact-finding mission has since said there are reasonable grounds to believe the United States committed a war crime in that strike. Washington has not publicly acknowledged responsibility.

Crucially, for the purposes of the current debate on AI, this was not an autonomous system deciding to kill. According to Bloomberg’s report of September 19 on the Pentagon’s investigation, flawed intelligence, outdated imagery, and an overreliance on AI-assisted targeting contributed to the strike. An analyst had reportedly flagged changes to the site in 2019, but the note never reached those preparing the strike. Bloomberg also reports that no one from the military’s civilian harm team reviewed the site before the strike. This was as much a human failure as a technical one. 

Whether what we have now is in fact artificial intelligence can be questioned, but the existence of fully natural human stupidity, unfortunately, is beyond doubt.

On intelligence and stupidity

AI chatbots can already perform feats that no single person can do. They can answer questions on pretty much any topic, in many languages, write everything from poems to code, solve complex math problems, and more. But it turns out that intelligence is rather difficult to define. And so, despite their obvious and growing capabilities, it is possible for very intelligent people, including top scientists, philosophers and entrepreneurs, to disagree on whether the AI systems we now have are in fact intelligent.

Stupidity, too, is not as easy to define as it may seem.

The Italian economic historian Carlo Cipolla wrote an essay, “The Basic Laws of Human Stupidity”, first published in Italian in 1976, in which he defined a stupid person as follows: “A person who causes losses to another person or to a group of persons while himself deriving no gain and even possibly incurring losses.” 

Alongside the stupid, he described a second type, the “bandit”, who gains while causing losses to others. Cipolla’s other two types are the helpless, whose actions benefit others at a loss to themselves, and the intelligent, who bring gains to themselves and others.

Given that the world seems to be owned and ruled by bandits, it is perhaps the unfortunate lot of most ordinary human beings everywhere to suffer losses helplessly. So far, those losses may have been tolerable, but the set of losses now threatened is not.

If AI kills many or most of us, that is not a tolerable loss.

Even if it does not kill us physically, it holds the potential to kill us financially. The risk of job losses due to AI is far closer than the threat of superintelligent rogue AIs. Talk of new jobs being created to replace the millions lost is just that at present – talk. No one can say what new jobs will be created, or how many.

Sceptics argue that the risk of superintelligence – an AI that is more intelligent than the best individual human minds in every cognitively relevant domain, and that may eventually become an AI system or network more intelligent than the sum of all human beings – is also just talk so far. There is nothing to suggest that its development is imminent.

However, the timelines for the projected arrival of superintelligence have been shrinking.

A 2022 survey of 738 AI researchers by the Machine Intelligence Research Institute of the University of Oxford produced an aggregate forecast of a 50 percent chance of “High Level Machine Intelligence” (HLMI) by 2059, down from 2061 in a similar survey held in 2016. HLMI was defined as unaided machines being able to accomplish every task better and more cheaply than human workers.

That’s not quite superintelligence, but it is close to AGI and would clearly bring about the jobs apocalypse… except that this survey was done before the release of ChatGPT in November 2022. The AI race only really took off after that.

Since then, experts from academia and industry have offered their own estimates for the arrival of AGI, and the estimates vary. Academics such as Prof Hinton and Turing Award winner Prof Yoshua Bengio have said they think it could be here in 10 years or less. Industry leaders offer similar or shorter timelines. Hassabis has suggested 5 to 10 years. Amodei predicted its arrival in 2026-27 and defended his prediction at the World Economic Forum in Davos earlier this year, saying he still thought it wouldn’t be “that far off”. Altman expects it to happen by 2030.  “Within the decade” has become a commonly voiced expectation.

The step from AGI to superintelligence would take some time, though probably not much. Musk has said that “AI may exceed the sum of human intelligence” in about 5 years. The whole idea of an intelligence explosion in machines relies on recursive self-improvement: AIs that can make better AIs, so that the pace constantly accelerates.

This dream – or nightmare – runs up against limits in the amount of available data, compute and energy. However, it may be possible to improve the harvesting of solar and nuclear energy in the next few years. Quantum computing could usher in a new era of computing power. Using AI-generated data to train future AIs makes them less smart over time. However, the quality of synthetic data itself is improving, and there is still plenty of untapped quality human data beyond the freely available content in big global languages such as English.

Even if the industry’s most bullish forecasts are discounted, and the limits on data, compute and energy hold for a while, the arrival of AGI within a decade or two, and of superintelligence not long after, seems quite possible.

Which brings us back to the question of whether such a superintelligence would then kill us all.

Why the warning matters even if the motives don’t

The answer is, it might. 

Firstly, it would have been trained on human data, and we human beings have really not treated one another or the other species with which we share our planet very well. Why should we expect an AI trained on our data to behave any better than us? Secondly, we would be the only threat to its existence, and we would be trying desperately to keep it under our control. If it develops an instinct for self-preservation, why would it want to have us around? Thirdly, we would compete with it for the resources it needs, such as electricity and water. Why not get rid of us and “optimise” itself by taking all the resources?

The risk is therefore definitely non-zero.

Long before then, humans misusing AI to spread disinformation, build weapons or target humans could create chaos. Deepfake videos and voice cloning already exist, and both have featured in cases of disinformation. The jobs apocalypse could undo social and political orders. The strain on fresh water in a heating world facing more extreme climate events, worsened by data centres, may produce unrest even in remote rural areas.

Suffice to say, the prospect of chaos resulting from developments in AI within one to three decades appears, at this point, quite plausible. AI safety clearly needs to be taken very seriously, and its definition does not start and end with existential risk and questions of alignment – whether the AI is aligned with good human values. It also covers the societal and systemic risks that could create such chaos.

That chaos could escalate to catastrophe, directly, through nuclear, chemical or biological weapons, or indirectly, through societal collapse or weaponised falsehoods. It could stem from an autonomous AI or from loony humans misusing AI, and the outcome would be as bad either way. It seems rather silly, then, to focus the entire argument on whether that is likely to happen in two decades or three – and to shrug it off with a “what, me worry?” if the answer turns out to be three decades.

It may be that the AI bosses are calling for a slowdown and regulation now for their own selfish reasons, such as boosting their IPOs or sustaining an AI bubble threatening to go bust. However, that would not necessarily make it the wrong thing to do. It would be a case of doing the right thing for the wrong reasons. And the time to act is now, not when some AI-turned-Terminator is knocking at the door.

Samrat Choudhury is an Associate Professor of Practice at Shiv Nadar University, Delhi-NCR.

Manoj Sharma is a Senior Director at Microsoft in Seattle. Microsoft has deep commercial ties to both OpenAI and Anthropic.

A quarter-century of Modi governance calls for questions and a hamper. To mark the occasion, we have offers for subscribers and other readers.