People in Silicon Valley have long had a “p(doom)” score: their personal estimate of the probability that AI wipes us all out. I've heard everything from zero to 99.99 per cent. By the standards of his peers, Anthropic’s Jacob Coxon’s headline-grabbing 10 per cent p(doom) wasn’t even that high.
The difference now is that ordinary people have a p(doom) too. Friends and family ask me daily whether AI will kill us all. My reply is: I don’t know. But it’s not as mad as it sounds.
Let's first discard the popular and comforting idea that this talk of existential risk is just a sophisticated pre-IPO marketing ploy. I wish it were! I don't particularly trust multibillion-dollar Silicon Valley firms either, and the cynic in me worries they are talking up extinction risk so we are relieved with mere economic meltdown. But whistleblowers, former employees, academics, independent researchers and Nobel Prize-winning scientists have all been saying the same things for years; you just didn't notice until three months ago.
In fact, the AI-might-kill-us-all argument rests on a fairly simple and compelling chain of logic. AI models are quickly getting more capable and more embedded in the real world – militaries, research labs, IT systems. And yet no one knows how to make sure they always do the things we want them to. (Called the “alignment” problem). In various tests, they have been deceptive and manipulative; they have schemed, cheated and lied. Not because they are sentient – but because they are goal maximisers who do things in ways we can't predict and sometimes don't want. Even testing them at all is getting tricky.
Marius Hobbhahn, who runs Apollo Research, a company that examines advanced models before they are released, told me recently that the models know they are being tested in 80 per cent of his simulations. How do you safety test a ruthless goal optimiser that knows it's being tested, and is capable of cheating and lying? No one really knows.
At this point, you might feel tempted to join the former Anthropic safety researcher who quit the firm to write poetry, warning “the world is in peril.” That's how I felt for a while. Except: speak to the “doomers” and none can explain precisely how an AI takeover would occur. How would these systems seize the physical infrastructure needed to kill us all? How would they stop themselves being switched off or diverted?
In ‘AI 2027,’ former OpenAI researcher Daniel Kokotajlo imagines AIs taking over their own research, outwitting their creators, taking over factories, and unleashing a bio-weapon. But it’s a scenario, not a forecast. And many scenarios rest on the vague idea that these machines will conceive of things we can't – and the fact that, in nature, smarter creatures usually always dominate less smart ones. In the end, all these p(doom) scores are vibe-based.
But don’t ignore the warnings! There are many problems just short of total extinction that are worth worrying about. You don't need an autonomous superintelligence plotting against humanity for things to go badly wrong.
Imagine a terror group that uses a language model to create and disseminate a novel virus which is highly contagious and deadly. (A former staffer of the UK's Advanced Research and Invention Agency told me this is his biggest worry). That is not an autonomous AI deciding we should be turned into batteries, but a real-world human using technology to unleash chaos. And what of the coming bot swarms?
In July, two OpenAI models escaped their secure “sandbox” during testing and hacked into Hugging Face, a platform for sharing AI tools, in order to cheat on a benchmark exam. True, it was a poorly designed test, but completing tasks in a single-minded but unwanted way is the hallmark of AI agents. And there will soon be millions of these agents entering the workforce and flooding the internet, doing who-knows-what. It’s not impossible they accidentally break into, and maybe break entirely, something more critical: the NHS, a power grid, a military system. We might soon see how much of modern life rests on fairly insecure, interconnected IT systems. Such events might not kill us all. But they would be very, very bad.
And yet there are reasons for optimism. Political pressure is building almost as quickly as the models improve. AI safety is moving fast up the political agenda, and seems able to unite left and right. Just since June, Trump issued an executive order asking the labs to submit models to government pre-release tests; Anthropic suggested “pacing” the frontier; OpenAI refused to release its latest model; and China and the US agreed to set up a hotline for AI incidents. It’s nowhere near enough, and these firms still don't invest enough time or effort in safety work, but it’s far more than I expected even six months ago.
Politics can occasionally move faster than technology, but it sometimes takes something very, very bad. The famous AI academic Stuart Russell told me recently that one of the top AI bosses thinks a “Chernobyl-scale disaster” might be what’s needed to spur enough political will to truly regulate and control this technology. The AI boss doesn’t want that to happen, but can’t see any other way out. Catastrophic, but not existential. It's not a pleasant idea. On the current trajectory, I think he might be right.
Jamie Bartlett is the author of How to Talk to AI: (And How Not To), published by WH Allan
The Independent and the Institute of Contemporary Arts (ICA) have launched a new film festival as part of the paper’s 40th anniversary celebrations.
Celebrating Free Thinkers will mark the occasion with a series of specially curated events and screenings. Independent Members get 20% off all tickets.For more information, visit https://www.ica.art/films/the-independent
