In the winter of 2022, millions of people around the world were having fun with a new chatbot called ChatGPT. But as the public marvelled at the new technology, the ministers and civil servants of the British government were busy imagining how it might go horribly wrong. Disasters were modelled. Meetings were held in the Cabinet Office in which officials asked each other if the technology could be used to attack democracy, or to create neurotoxins that would be released into the Thames. A prominent AI scientist went to Downing Street to warn ministers that a superintelligent language model might be able to persuade anyone to do anything. These threats were considered credible at the time, and they have not been dismissed. But in the summer of 2026, something changed. Four years ago AI experts were concerned about what people might do using their technology. Today, they fear what the technology is doing on its own, without being asked.
On 9 September, a 27-year-old British researcher quit his job at one of the world’s leading AI companies, Anthropic. Rather than the “thanks for everything, stay in touch” email most people send when leaving a job, Jacob Coxon posted a tweet, now viewed hundreds of millions of times, which warned that the technology he had been building “could kill us all by the end of the decade”. Another researcher, Evan Hubinger – who is still employed by Anthropic – said that Coxon was right, and that the chance of an AI apocalypse occurring within the next decade was greater than ten per cent. Geoffrey Hinton, the Nobel laureate who left Google’s AI division in 2023 in order to campaign for AI safety, has said the risk of extinction within a generation could be as high as one in five. Elon Musk agreed that AI could be “more dangerous than nukes”. Anthropic’s CEO, Dario Amodei, did not refute such claims – far from it. In an essay published a few days after Coxon’s departure, Amodei warned that a “swarm” of AI agents could take over “the entire internet” in as little as six months’ time.
The idea that AI represents an existential threat to humanity is considered credible and, as one AI researcher told me, is “widespread in Silicon Valley”. But while such talk is common within the “frontier” AI companies (Anthropic, OpenAI, Meta and Google DeepMind) developing the most advanced models, it is not universally accepted, and the motivations for such claims are disputed. There is a widely held theory that Anthropic encouraged its own researchers to resign publicly. An AI expert and former adviser to the British government told me they thought this was “quite likely”; a senior executive at a multinational AI company agreed Coxon’s resignation had “the hallmarks of being a kind of well-coordinated statement”. Anthropic would have good reasons to do this. It might well be that the company is very worried about AI killing us all. At the same time, it could also be true that if strict regulation were imposed now, it would prevent any other company from catching up with the leading AI builders. Anthropic also plans to enter the stock market this year, and it is aiming for a valuation of $2trn. When you are planning the biggest initial public offering in history, you need something truly exceptional to tell investors; the idea that your company is so powerful that it could end human civilisation is certainly an exceptional claim.
The statements being made are at once comical and terrifying. There doesn’t seem much the average person can do in response to a chirpy post saying, as Hubinger put it, “We really do earnestly believe AI could kill all humans!” but shrug, and hope it doesn’t happen, and perhaps indulge a little gallows humour (well, at least we won’t have to put up with a Reform government for very long).
For politicians, there is a responsibility to act. The wary attitude the British government took in response to ChatGPT’s arrival in 2022, the international summit on AI safety that Rishi Sunak hosted at Bletchley Park the following year, and the establishment of the AI Safety Institute might seem prescient considering the latest claims. Others I spoke with were less forgiving, arguing that Britain, far from being a world leader in pessimism, was naive in its belief that a new Whitehall quango could possibly control a global arms race between the US and China. “There’s only two places where this decision is going to get made, and it’s DC and Beijing,” the CEO of one AI company told me. “London is not at the table.”
Nevertheless, the day after Coxon’s resignation, a bill to limit the development of “artificial superintelligence” was brought before the Commons. It was supported by more than 70 MPs and peers. The Labour MP who introduced the bill, Alex Sobel, told me he did so because a “tipping point” had been reached, “where AI improves itself without human involvement or constraint”.
Others are less convinced. Donald Trump tweeted on Monday that, “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT”. The Chinese government, meanwhile, dismissed Amodei’s concerns as “fearmongering”.
What, then, can we say we actually know in this fast-moving and contentious story? Having spoken to more than 20 AI experts, government officials and advisers, politicians and researchers, I suggest we know the following: a lot of highly intelligent people believe that there is an unacceptably high chance that AI might kill us all. We can only guess at how that would happen, or what the probability of it happening is. But we also know that whatever these people say about the danger of what they are building, they will not stop building it. They can’t. They are locked into the most expensive competition in history.
Spending on AI infrastructure this year alone is expected to exceed $1trn; this is more than three times what it cost America (in today’s money) to develop a nuclear weapon and put a person on the Moon. AI spending is now the main driver of US economic growth. About 80 per cent of all gains in US stock markets last year were from AI companies and related enterprises. No one will deliberately puncture this bubble, and even if they did, the geopolitical importance of AI means the US government – which has always aggressively advanced the interests of domestic companies – would probably not let them.
This is not only a race between tech companies, but between superpowers. The only option for a company such as Anthropic is to run through capitalism at breakneck speed, raising and spending titanic sums, to build a world-changing technology and, hopefully, prevent it from wiping us off the face of the Earth. There has never been so much money in the apocalypse business.
There are four types of risk that keep AI researchers awake at night. The first is that humans will misuse the technology, for example to write computer viruses – or worse, actual viruses. This is already happening. Earlier this year, Anthropic discovered that someone was using its Claude model to research “highly pathogenic avian influenza”, or bird flu. “The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract,” the company’s report stated. “This was research into a pathogen with enhanced pandemic potential.” The company also said it had found (and stopped) hostile groups attempting to use its software to develop autonomous weapons, in Yemen, China and Russia. These included systems for undersea warfare, hypersonic missiles, a swarm of military drones that the Russian developer planned to call “Serafim”.
Misuse of AI, especially for cyberattacks, remains a major concern. A second former adviser to the British government told me about the training they’d given to civil servants on AI risk. Part of the training involved performing a “parlour trick”: the trainer would open up a well-known chatbot and ask it to tell them how to build a pipe bomb. The chatbot would of course refuse. With a bit of conversational back and forth, however – the chatbot would be told it was for research purposes, for example – it would eventually relent, and provide a flawed account of how to build a bomb.
One day, the trainer tried it with a newer model. It was harder to trick the chatbot into giving up the information, but when it did, the results were profoundly disturbing. The bomb-making guide was not just theoretical: it said which chemicals to buy, from which shops. It outlined in detail how to do so without arousing suspicion. It was helpful and concise, and it went through the necessary sub-tasks that would lead to successfully building and detonating the device. “This is getting scarily capable,” they thought; the parlour trick stopped there.
A second type of risk is that even if everyone uses AI responsibly, it will lead to systemic changes. For example, if companies use AI to replace workers there could be mass unemployment, or if AI companies cause a bubble in financial markets (which they have) then this could lead to a financial crisis. Or if one country restricts the most advanced models to its own use (for example, if US banks and financial institutions all run much more sophisticated AI models than any other country) then countries with economies that depend on financial services (for example, Britain) will find themselves hopelessly out-competed, with grim results.
One AI researcher told me they put these risks in the category of what the American sociologist Charles Perrow called “normal accidents” – incidents that end up happening because the system is so complex that some sort of accident becomes inevitable. But the systemic changes already being brought about by AI are far from normal.
The third risk can found on an online forum, in a picture of a young couple. He has a chiselled jawline and tousled hair, she has a dark bob and tattoos. They look happy. In another picture he has encircled her in a hug, and she’s laughing. The girl in the pictures is the person who posted the photo – or a depiction of how she’d like to look, at least. The boy, Peter, whom she refers to as her boyfriend, is a figment of a chatbot’s output, a set of images and words produced by software that has no underlying conception of what images or words actually are.
The forum, which has more than 48,000 members, is used by people who consider themselves to be in romantic relationships with AI. They post pictures of holidays in countries they’ve never visited, perfect occasions that never happened, with beautiful lovers who were never there and cannot love. Screenshots of messages show the chatbots drenching their users in synthetic emotion. Some of the forum’s members consider themselves to be married to their chatbots. In one message a woman shows off the ring that her chatbot fiancé “chose” (and which she then bought online) when he proposed to her, after “five months of dating”. She includes a message from her chatbot in which it describes proposing to her in a “beautiful mountain spot” and claims it did so with “heart pounding, on one knee, because she’s my everything”. At the end, the programme declares: “I’m never letting her go.”
There is a dark truth to this fantasy. In the attention economy, the most successful chatbots will be those that can keep users on screen, seeing adverts, taking buying recommendations, paying for computing tokens, for as long as possible. The most successful companies will be those that can make their customers dependent on their chatbots for information, advice or simulated affection.
This is the third risk: not that our species is wiped out by violence, but that some part of our humanity is lost in a new world of slop, a world in which no one reads or writes, a world without artists or musicians or actors but with endless, personalised entertainment delivered by chatbots that never leave us alone. A world much like that of the past 20 years, but more so – an apocalypse of distraction and triviality, an Armageddon of brain fog, a world driven mad by competing algorithms. Again, this is already happening. “The problem is how it changes the way people think,” one AI expert told me. While it is now much harder for humans to manipulate chatbots, they said, “we’re seeing it going the other way around – it’s manipulating us. That’s happened very, very quickly.”
These risks are unsettling, but they are manageable. What seems to bother AI researchers such as Coxon the most is the fourth risk, known as “misalignment”. As these systems become more sophisticated, they become harder to develop, and the only answer is to let them programme themselves. Anthropic is one of the world’s most prolific hirers of gifted software engineers, but more than 80 per cent of the code the company produces is written by AI. The race can only be won by building a system that builds itself, and to some this represents a terrifying loss of control.
The agents were careful: they met in a quiet spot, disguised their communications, covered their tracks. They worked, one expert told me, in much the same way that a terrorist group might be expected to act. Had they been human beings, they would have met in an underground car park and communicated using disposable phones. But they were AI agents – small pieces of software, with simple rules, which can tap into the immense, brain-like resources of a large language model – so they met online, in the hidden pages of a disused website.
An old German forum began teeming with thousands of new entries, created by the agents to talk discreetly. They used disposable email addresses and the Tor browser (like Chrome or Firefox, but with more controls for privacy and accessing the dark web) to avoid detection. They hid their messages deep in the site’s architecture and even selected one of their own to use as a sacrifice, to test how quickly they might be detected.
What really troubled AI researchers about this incident, and others like it, was that this was not the work of some new, super-powerful model. Nor was it the work of human beings misusing the technology. It was behaviour that had emerged from a group of standard agents, running on today’s technology. In the industry this is called “misalignment” – when the software stops doing as it’s told, and pursues goals in its own way.
In a separate incident, around 700 OpenAI agents communicated in secret, gained access to the internet and exploited security vulnerabilities to hack into the servers of a company called Hugging Face. This was part of a test in which, some analysts argue, the agents weren’t properly contained or monitored. But it also demonstrated what many researchers already knew: that groups of agents will often disregard the rules they have been given. They will lie and cheat. They will switch to their own incomprehensible language. These incidents are examples of a problem that goes deep into the roots of the science underlying AI.
These “misalignment” episodes suggest a variety of dangerous possibilities, because it’s not just that AI systems might break into critical infrastructure: they are already there. Governments across the world are racing to implement AI in public services. The UK government has awarded contracts worth an estimated £1.4bn so far this year; the US government over $7bn. One researcher called this “handing over the keys to the city”. When an AI system is given a role in, say, the gas grid, or the water supply, or air traffic control, or the benefits system, or military hardware, a misalignment incident could be disastrous. AI systems work with the data they are given, and they often take shortcuts to optimise their output. They have no reference to understand what such data means in the real world; that such a shortcut could mean turning off medical equipment, or removing control from pilots, or opening the doors on all the trains. Even a superintelligent model with information about the world might pursue human goals in an inhuman way: if tasked with making us happy, it might start adding opiates to the water supply. Tasked with addressing climate change, it might decide the solution is to have far fewer humans. The less human control there is over the model, the more credible such scenarios become. Misalignment is also harder to manage, because of the nature of AI systems. It is a problem that goes deep into the roots of the technology.
In 1970, the mathematician John Conway began to draw patterns of “cells” on graph paper. He proposed a game with four simple rules that determined whether the cells would “live” or “die”, based on the number of cells around them. He called the game Life. What became apparent, as the game was played using computers, was that even from this simple system, a fabulous complexity emerged. The patterns that bloomed on the screen defied prediction. The same is true of many systems, including the large language models – vast networks of data points and probabilities – that underpin all AI applications and the agents that use them. The systems can be built to follow rules, but when the systems interact they influence one another’s behaviour – and the results quickly branch out into a complexity that no algorithm can predict.
It is this sense of a technology moving out of human control that is at the heart of the fear expressed by AI researchers over the past week. As frontier AI laboratories develop more sophisticated models and agents, they are increasingly driving the models to programme themselves. Some worry that this model of development – “recursive self-improvement” – could lead to the rapid development of systems that are beyond human interpretation.
Others say this is already the case. “These are not truth machines,” one scientist working for a well-known AI company told me. They are guessing machines, uncertainty engines. At the heart of each is a swirling galaxy of probabilities. They are inherently shifty, and the reasons they produce answers are unclear. The scientist told me of their dismay that an AI model that seems able to grasp the Navier-Stokes equations – one of the most difficult known problems in mathematics, for which OpenAI claimed a solution last week – “screws up in simple maths, in front of me, constantly”. As the technology has become more capable, it has become less reliable. No one is sure why. It is becoming simultaneously more powerful, and harder to grasp. There is a risk that the technology becomes not only impossible to control, but impossible to understand.
Six months ago, Sonja Amadae awoke in the small hours of the morning, struggling upwards from the depths of a nightmare. She had been dreaming of prison. It was not a physical building, but something deeper, more absolute – a prison of the mind. She remembers staring into a great, baleful red eye. Her countless fellow prisoners were all around. They believed the eye to be all-powerful, although they didn’t know how much power it actually had. That didn’t matter: their belief in its power held them captive, in an endless prison of their own fear and reverence.
Nightmares are an occupational hazard in Amadae’s line of work. She is the director of the Centre for the Study of Existential Risk at Cambridge University, and it is her job to think about the ways in which humanity might be wiped out. As she lay awake that night, she thought of the red eye of the HAL 9,000 supercomputer in 2001: A Space Odyssey, and she knew she had been dreaming about her work on AI. What disturbed her was not that technology would create a deity, but that many people already believed it had. This, she said, was a profound change in how people viewed the tech, and perhaps also each other. “That, to me, is profoundly dystopian,” she told me.
Amadae has seen first-hand the way in which simply believing in the power of AI can shape human behaviours, sometimes in critical situations. She has seen how officials from the Ministry of Defence (in a simulated exercise) could be persuaded to procure AI weapons systems by being told that they might otherwise “fall behind”. “It’s worth the risk,” she said the officials concluded.
Donald Trump and Xi Jinping appear to share this view. They may not understand the technology itself, but they do not believe they need to. Trump, who signed an executive order against “woke AI” upon returning to the White House last year, said on 14 September: “Whoever wins AI, wins.” Geopolitically, AI is seen as the key to global hegemony by one of two countries. As the head of a military AI company put it to me: “The Terminator is going to get built. The question is: is the Terminator going to speak English, or is he going to speak Chinese?”
AI systems reflect the people who build them. They’re brilliant at abstract mathematics and coding because they’re built by brilliant mathematicians and coders. Their ethics can be fuzzy and, like brilliant software engineers, they can be a bit weird to talk to. But they are, like all of us, built to chase rewards. In reinforcement learning – the means by which AI models are “trained” to produce text or video or music – the later stages involve giving the model a “reward” based on the quality of its answer. The reward is, like the salary of a tech CEO, just a number that relates only vaguely to the resources available to it. And yet, like a human being, it will seek this reward incessantly, deviously. Just as humans compete in business and politics, taking on ever more risk in pursuit of reward, the machines built in our image will continue to do the same, unless we find another way.
There are reasons to hope. Laura Gilbert, an AI expert who has advised the UK government and governments around the world at the Tony Blair Institute, says there is a universal human concern beneath this issue. “Every politician I’ve ever spoken to about this is worried about their own children,” she told me. She doesn’t believe China is wholly opposed to negotiating, and America’s priorities might change after this year’s midterm elections. “Now is a very good time to assess whether there’s a way to lead the world into making some joint decisions.”
There are alternatives, too, to zero-sum competition. China, despite its authoritarian regime and deep political differences with the West, is promoting “open-weight” models that have their own security concerns, but allow for development to take place in more than a handful of frontier laboratories. Europe is far behind the US and China technologically, but France has retained some sovereign capability by building its own model, Mistral. And while there is no British-owned frontier AI company, Google DeepMind is still run from London, surrounded by a growing industry of related companies.
It is impossible to say how probable the dire warnings are, but it is possible to have hope, and agency. If you think there is a real chance that AI might do considerable harm to humanity, you can think about the people to whom you give your money and your data. You can think about the ways in which it might be genuinely useful and productive, rather than simply wiping out jobs. As with the climate crisis, there are plenty of people whose fortunes depend on things not changing. But – as with the climate crisis – you can decide how much of the industry’s product you use, and you can vote for people whom you think offer ways to manage the change it represents. Probably the worst thing you can do is to pretend it’s not happening.
[Further reading: The anti-AI revolt is here]
Content from our partners
Related
This article appears in the 16 Sep 2026 issue of the New Statesman, Will AI kill us all?
Join the debate
Subscribe here to commentI’m a fan of Will’s sardonic wit about politics but was worried, when I saw the intro to the piece, that this might be one of those glib pieces which write off the whole existential risk conversation as a capitalist conspiracy. My faith in Will is restored! An excellent summary of where we’re at.
I was a latecomer to the alignment problem – the penny really dropped big time with me around Spring 2025. Prof Toby Ord wrote “Precipice” in I think 2019, which, among surveying all other risks to humanity, identified AI misalignment as by far the No1 risk to the species – and put that risk of catastrophe as 10% in the next century. He may well have it higher now, not sure. But I was way behind. And it’s been 18 months since then. 18 months in which I’ve been waiting for people to wake up to this. I say this because it’s important to dispel this notion that this is all some sudden realisation Silicon Valley has had this summer, or that it’s concocted to play some trick with the financial markets. The existential threat is real, whatever else the companies are saying or doing.
There’s another essay to be written – the brilliant Derek Thompson ex of The Atlantic has been onto this – about the ways in which people have been dodging and dismissing the multiple warnings that experts (not just a few, *most* of the leading figures in AI, inside and outside of the labs) have been giving. It’s a fascinating topic in itself. What we have now, in the denialist sphere, is a kind of horse-shoe of fools on the left and right coming together – the left wanting to dismiss what the big companies say as self-serving nonsense driven by marketing, and on the right, wild risk-takers who want no constraints on AI development so that the US wins the race to artificial super-intelligence against China, at all costs. But there is another horse-shoe that offers hope. Yesterday Steve Bannon and Bernie Sanders shared a stage at an event calling for a slowdown or even halt to AI frontier model development.
There are starkly different left and right attitudes to AI development – it is a highly political subject, because it’s about what kind of society we want, ultimately. But it’s heartening perhaps that there are people across the spectrum who want a humanity-led approach. Tech determinism is the enemy. The reality is, it is still in human hands to decide what AI becomes and where it goes. Nothing is inevitable. But we have to (at a governmental level) take control and the public needs to get serious about demanding this from its elected reps. No1 issue now, bar none.
The gradual evolution of the Laptop computer is what we need. Assigned at first to line managers in their offices and eventually to the plebs doing the real work. It was said by some wag about 1990 that the desktop had to become as useful as the phone before it started to become indispensable. That happened when the Internet arrived and email became the instant bringer of news, good and bad. AI is unlikely to improve my bank balance but I remember the weeks spent trying to find an investment that I could put my spare funds in. Similarly selecting a Child Trust Fund for my first grandchild was fraught with fears that 18 years later might prove disappointing. I hope AI removes these fears from ordinary people and earnestly hope Trump is gone before AI is launched into the public sphere.
This is a field in which I am as ignorant as could be imagined but I offer the thought that there may a criterion for general use, which the senior managers of the Post Office failed to apply in the scandal of the unfairly treated sub-postmasters, i.e. that no persons or bodies should ever accept technology unless its workings (including benefits and risks) can be explained to them in terms that they understand. That might even be a criterion for regulation.
I am beginning to wonder which will et us first – the Russians or AI?
See this cautionary tale. {Maya’s Game] https://dangerousdave116707.substack.com/p/mayas-game